acceptodds
Under review as a conference paper at ICLR 2027

Delta-GRPO: Integrating OPD-Style Policy Shift into Reinforcement Learning

Abstract

On-policy distillation (OPD) transfers teacher knowledge through token-level feedback on student-generated responses. Recent work shows that the change between a teacher’s policies before and after reinforcement learning can also provide a useful learning signal. In contrast, group relative policy optimization (GRPO) assigns the same outcome-based advantage to every token in a response, without explicitly using token-level policy changes for credit allocation. We propose a method that combines OPD-style guidance with GRPO’s outcome-based optimization, using the training policy’s accumulated shift to guide token-level credit allocation during reinforcement learning. Across eight mathematics benchmarks, the method outperforms the evaluated RL and OPD baselines with Qwen3-4B and Qwen3-8B. It improves macro accuracy over GRPO by 13.59 and 12.04 percentage points, respectively, and over SC-GRPO by 8.90 and 7.89 points.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, on-policy distillation, large language models

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_qbaVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a reinforcement learning method called Delta-GRPO. By leveraging the policy shift accumulated by the model since its initial state, this method enables a redistribution of credit—refined to the token level—that aligns with the final outcome within the GRPO framework.

Critical strengths

It integrates "token-level policy-shift" information into the GRPO framework, eliminating the need for an external teacher model or an additional distillation stage. The formula is intuitively designed while preserving the mean and sign characteristics of the original "response-level advantages."

When applied to Qwen3-4B and Qwen3-8B models, Delta-GRPO substantially outperforms GRPO and other baselines across eight mathematical reasoning benchmarks. Notably, in terms of macro accuracy, it achieves improvements of 13.59 and 12.04 percentage points, respectively, over GRPO.

The empirical analysis is comprehensive, covering component ablation studies, investigations into "modulation strength," Pass@k evaluations, and an analysis of training dynamics. The ablation results confirm the importance of both "outcome alignment" and "response centering."

Critical weaknesses

Several closely related methods—such as Direct-OPD, OPD², EP-GRPO, and SC-GRPO—have already explored policy divergence signals or credit reweighting at the token granularity level. The proposed approach appears to be merely an incremental combination of existing ideas, and the distinction between it and EP-GRPO is not clearly defined.

Changes in token probabilities do not necessarily serve as an accurate reflection of a token's contribution to the correctness of the reasoning process. The paper lacks compelling ablation studies—such as shuffling the sequence of policy shifts or employing random, mean-preserving token weights—to demonstrate that the observed performance gains genuinely stem from meaningful historical information.

Delta-GRPO increases computational time per training step by approximately 55.6%; however, the primary comparisons presented in the paper do not account for the total computational budget used during training. Furthermore, the weak performance of EP-GRPO raises questions about whether the chosen baseline implementations are sufficiently competitive.

Delta-GRPO significantly reduces token entropy and increases the proportion of sample groups with zero reward variance, raising concerns about a potential decline in the model's exploratory capacity. Moreover, the experiments are limited to two Qwen3 models and mathematical reasoning tasks, leaving the broader applicability of the method unclear.

Questions for authors

see weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer qbaV [Part 1/3]

Dear Reviewer qbaV,

Thank you for your careful review and constructive feedback. We appreciate your recognition that the method integrates token-level policy-shift information into GRPO without an external teacher or distillation stage, and that the empirical analysis covers ablations, modulation strength, pass@k and training dynamics. We ran the ablations you proposed and address each weakness below.

1. Distinction from Direct-OPD, OPD², EP-GRPO and SC-GRPO (W1)

Delta-GRPO is the only method in which the outcome advantage fixes both the sign and the total credit of each response, while a token-level signal from the learner's own training history decides only where that credit goes.

Table R1. Comparison with related methods.

MethodToken-level signalExtra teacher / stageRole of the token signalResponse mean & sign of advantage
ExOPDTeacher vs. reference log-ratio (extrapolated)Yes / yesReplaces the outcome reward (distillation)n/a
Direct-OPDTeacher post-RL vs. pre-RL log-probYes / yesReplaces the outcome reward (distillation)n/a
OPD²Reasoning-tuned teacher vs. its base, centered over candidate tokens at a prefixYes / yesReplaces the outcome reward (distillation)n/a
SC-GRPOKL between solution-conditioned and unconditioned policyNo / noSuppresses updates at low-divergence tokensNot preserved
EP-GRPOToken entropy and policy/reference log-ratio, normalized in entropy-progress bucketsNo / noEntropy-gated rescaling of the advantageNot preserved
Delta-GRPOLearner's own πold\pi_{\rm old} vs. π0\pi_0 on the sampled tokenNo / noRedistributes the outcome advantage across positions of each responsePreserved exactly (Prop. 1)

Table R2. Delta-GRPO vs. EP-GRPO.

EP-GRPODelta-GRPO
What sets each token's weightCurrent entropy (gating) and policy/reference divergenceAccumulated shift from initialization, aligned with sign(Ai)\mathrm{sign}(A_i)
Normalization unitCumulative-entropy progress bucketsPositions within one response (centered)
Response's mean advantageChanges with token statisticsUnchanged by construction
Response whose signal is constantStill rescaledReduces exactly to GRPO

These choices matter empirically: removing alignment drops 4B macro accuracy from 66.39 to 55.15, and breaking the token-to-history correspondence drops it to 49.63 (Table R3). Under the same setting, SC-GRPO reaches 57.49 and EP-GRPO 42.10.

AuthorsOpenReview ↗

Response to Reviewer qbaV [Part 2/3]

2. Shuffle and Random Mean-Preserving Ablations (W2)

We ran the shuffle ablation you propose. Removing the token-to-history correspondence drops macro accuracy from 66.39 to 49.63, below GRPO.

Theoretical safeguard. The external answer verifier supplies fresh correctness feedback on current rollouts, which, together with length shaping, determines the group-relative advantage. Historical shifts redistribute this advantage through positive multipliers within 1±ηk1\pm\eta_k (ηk≤0.05\eta_k\le0.05), preserving its sign and response-level total while changing token priorities (Proposition 1, Eq. 6).

Table R3. Qwen3-4B, average accuracy (%).

MethodMATH500AIME24AIME25AMC23MinervaOlymp.HMMT25AIME26MacroΔ vs. GRPO
GRPO92.1041.3535.3187.1143.8467.0618.8536.7752.80—
No alignment93.2746.4638.6587.8944.3968.4321.1540.9455.15+2.35
Shuffled shift (new)90.2233.3630.0882.5441.9162.1720.0336.6949.63−3.17
Delta-GRPO95.5066.8855.2196.1746.2374.8736.9859.2766.39+13.59

The shuffled multipliers are themselves random, mean-preserving token weights: each response's weights have mean one and are assigned to tokens at random. They also match the exact multiset of the full method, so the shuffle isolates the token-to-history correspondence from any change in the weight distribution. Without that correspondence, the method falls below the GRPO baseline it modifies.

Token-level case studies (Appendix E.3, Figure 8). Figure 8 visualizes mi,t−1m_{i,t}-1 token by token across four responses (two correct and two incorrect), with red/blue indicating multipliers above/below one. In the two correctly solved, positive-advantage cases, self-verification windows (e.g., "But wait, let me think" and "But wait, maybe I need to think more generally…") have mean multipliers of 1.00345 and 1.00407, versus 0.99680 and 0.99749 for the respective final-answer windows. Relative to uniform GRPO weighting, this redistributes positive credit toward self-verification reasoning and away from final-answer tokens.

Manuscript Revision. We have added the shuffled-shift row to Table 2 (§5.2).

3. Training Compute and Baseline Competitiveness (W3)

(a) Compute. Delta-GRPO reaches better results in fewer steps and stays ahead as training continues. On online MATH500, it exceeds GRPO's final step-140 accuracy at step 60 (92.03 vs. 91.80) and reaches 93.98 at step 140. Checkpoints taken well before GRPO's 20.668 training hours already exceed GRPO's final checkpoint:

Table R4. Online validation accuracy (%) within the GRPO wall-clock budget, Qwen3-4B.

BenchmarkGRPO, step 140 (20.668 h)Delta-GRPO, earlier checkpointΔ
MATH50091.8092.20 (step 80, 15.796 h)+0.40
AIME2441.8846.04 (step 70, 12.755 h)+4.16
AIME2533.1336.77 (step 70, 12.755 h)+3.64

The training reward, logged at every step, is also higher at nearly matched elapsed training times: 0.3232 for GRPO at 20.668 h (step 140) vs. 0.3342 for Delta-GRPO at 20.471 h (step 96).

AuthorsOpenReview ↗

Response to Reviewer qbaV [Part 3/3]

Table R5. Total training cost and accuracy, Qwen3-4B (140 updates each).

ItemGRPODelta-GRPOExOPDDirect-OPD
Teacher preparation——20.668 h20.668 h
Main / student training20.668 h32.168 h12.953 h13.242 h
Total20.668 h32.168 h33.621 h33.910 h
Macro (%)52.8066.3953.2649.84

(b) Baseline competitiveness. SC-GRPO, implemented under the same setting, improves over GRPO by 4.69 and 4.16 points at 4B and 8B. Our EP-GRPO implementation follows the original method and EP hyperparameters exactly (λ=0.1, η=0.2, ten progress buckets, γ=5). Its weak result stems from non-terminating generation: EP-GRPO policies fall into repetitive output loops and fail to stop, so responses are truncated at the length cap (at 8B, 80.73% of endpoint samples reach the 16,384-token cap). The original paper operates with 2,048-token generation (Qwen2.5-3B/7B), where this behavior is cut off early.

Manuscript Revision. We have added the matched-time and total-cost comparisons, and EP-GRPO diagnostics, to Appendix C.3 (Tables 9–11).

4. Exploration and Generality (W4)

Exploration. If lower entropy reflected collapsed exploration, Delta-GRPO's gain would shrink toward zero as kk grows. Instead, at 4B the gain persists at pass@32 and is often larger than at pass@1:

Gain over GRPO (pp)AIME24AIME25HMMT25AIME26
pass@1+25.53+19.90+18.13+22.50
pass@32+16.67+20.00+26.67+23.34

pass@k is higher on all eight benchmarks at both scales (Tables 12–13). The larger share of zero-variance groups (36.72% vs. 21.88% at step 70) includes groups in which all responses are correct, consistent with faster growth of training correctness (Figure 5).

Generality. We added a larger model and a different task domain.

Table R6. Qwen3-32B, average accuracy (%); protocol as in Table 1.

MethodMATH500AIME24AIME25Mean
GRPO93.6549.0636.6759.79
Delta-GRPO96.0271.9458.3375.43
Gain+2.37+22.88+21.66+15.64

On the same three benchmarks, the gain is +16.28 at 4B and +14.92 at 8B, so it does not shrink with scale.

Table R7. Code, Qwen3-4B trained on mathematics only; same evaluation protocol.

BenchmarkGRPODelta-GRPOGain
HumanEval81.7184.15+2.44
HumanEval+74.3977.44+3.05
MBPP69.3172.49+3.18
MBPP+61.1162.17+1.06
CRUXEval-O45.5046.13+0.63
Mean66.4068.48+2.07

Although trained only on mathematics, Delta-GRPO improves all five code benchmarks.

Manuscript Revision. We have added the Qwen3-32B and code results to Appendix D.5 (Tables 17–18), and the pass@1 versus pass@k comparison to Appendix D.1 (Table 14).

We hope these clarifications and additional results address your concerns. We welcome any further questions or discussion.

AuthorsOpenReview ↗

A Kind Reminder for Discussion

Dear Reviewer qbaV,

Thank you again for your time and constructive comments on our work.

We have submitted our detailed response, which we hope will address your initial concerns. As the author–reviewer discussion period is coming to an end, we respectfully ask if you could review our updates at your convenience.

If you find that our response has adequately resolved your concerns, we would be deeply grateful if you might consider reflecting this in your final evaluation. We would be happy to provide any further clarification if needed.

Best regards,

The Authors

Reviewer_UtxcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Delta GRPO uses the learner's accumulated change from its initial policy to redistribute a response's outcome advantage across tokens. The log probability shift is clipped and normalized, aligned with the sign of the response advantage, and centered over the response before constructing a bounded multiplier. This preserves the mean and sign of the response advantage while changing the token priorities. The evaluation covers two Qwen3 sizes, eight mathematics benchmarks, component ablations, solution coverage, and training cost.

Critical strengths
  1. The main empirical improvement is substantial and unusually consistent across the benchmark suite. At 4B, macro accuracy rises from 52.80 for GRPO and 57.49 for SC GRPO to 66.39. At 8B, Delta GRPO reaches 67.45 versus 59.56 for SC GRPO. The principal GRPO comparison uses matched initialization, prompt order, training settings, and three training seeds. These results deserve more weight than a small gain on a single competition set.

  2. The construction in Equations 2 to 4 is simple and constrained in useful ways. A positive multiplier cannot reverse the outcome signal, and response centering prevents the method from merely changing the average advantage of an entire response. Table 2 tests both alignment and centering rather than assuming that these safeguards automatically explain the gains. Removing alignment has a particularly large effect, while the more modest centering benefit is reported without claiming dominance on every benchmark.

  3. The evaluation distinguishes individual answer accuracy from the probability of finding at least one correct answer in a sample bank. The discussion of the one additional solved question at 8B on AIME24 is appropriately cautious. It is also helpful that the paper discloses the 55.6% increase in measured step time and reports expected output token budget comparisons rather than presenting the method as computationally free.

Critical weaknesses
  1. The ablations establish the importance of processing the shift in a particular way, but not yet the value of its historical content. A policy becoming more confident in a token does not necessarily mean that the token deserves more credit for solving the problem. Table 2 changes alignment and centering while retaining the history signal; neither control tests whether the token to history correspondence matters. A useful comparison would shuffle the shift values within each response while preserving their distribution, mean centering, and alignment. This is especially relevant because Section 5.5 reports multiplier mass very close to uniform despite the large final accuracy gains. The result is plausible, but the claim about better credit allocation needs evidence beyond the algebraic preservation properties.

  2. The compute comparison supports a more limited conclusion than the closing discussion suggests. Appendix C.3 measures 827.2 seconds per Delta GRPO step versus 531.5 for GRPO, and the expected output token analysis concerns inference rather than the training budget. Reaching a particular MATH500 accuracy in fewer updates is encouraging, but does not settle whether the final eight benchmark advantage survives equal elapsed training time. A matched time comparison for the principal pair would answer that question directly. The current results clearly show better endpoint quality at matched updates; the overall quality versus training compute claim remains less settled.

  3. The tested optimization regime is narrower than the GRPO formulation may initially suggest. Appendix C.1 states that each rollout batch produces one optimizer update, the ratio denominator in that branch is the detached current policy, and recorded clipping fractions are zero. Thus the empirical results do not exercise meaningful clipping or policy staleness across repeated updates on a batch. Since the method changes token advantages inside a clipped surrogate, its behavior under rollout reuse is an important boundary. A small experiment with more than one update per collected batch would be useful, or the paper should make the present scope explicit near the objective rather than leaving it to the implementation appendix.

Questions for authors
  1. Table 3 stops at a modulation strength of 0.05, which is also the best setting tested. Was this value selected using separate validation data or the reported benchmark suite? The exploratory sweep shows a clear upward trend; a modest extension would clarify whether 0.05 is near a useful operating region or simply the last point tested.

  2. Please show the principal pair's per seed macro scores in addition to the question bootstrap intervals. Three training seeds are already available. Their dispersion measures a different source of uncertainty from resampling evaluation questions, so both would be useful.

  3. The weakest adapted baselines require a little more interpretation. EP GRPO ends below the base model in Table 1. Appendix C.3 explains the adaptation, but a compact original setting versus shared setting check would help establish whether the method is intrinsically unsuitable here or sensitive to the implementation choices. This is secondary to the much stronger matched GRPO and SC GRPO comparisons.

  4. For the OPD baselines, separate the teacher preparation cost from student training cost in the comparison table. The appendix already states that they use a frozen GRPO teacher and a different rollout count. Making those differences visible beside the results would prevent readers from treating all rows as equal budget alternatives.

  5. Figures 1 and 2 should match the reward definition used in training. Figure 1 equates correctness with positive relative advantage, and Figure 2 shows zero or one rewards. Table 6 and Appendix C.1 instead specify positive or negative one correctness rewards plus a length penalty, and explicitly note that correctness need not determine the advantage sign. The diagrams should distinguish correctness from shaped relative reward.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Utxc [Part 1/3]

Dear Reviewer Utxc,

Thank you for your careful review and constructive feedback. We appreciate your recognition of the consistency of the gains across the benchmark suite, the constrained design of Eqs. (2)–(4), and the separate reporting of accuracy, coverage and cost. We ran the controls you proposed and address each point below.

1. Value of the Historical Content: Shuffled-Shift Control (W1)

We ran the control you specify. Removing the token-to-history correspondence lowers 4B macro accuracy from 66.39 to 49.63, below GRPO's 52.80.

Table R1. Shuffled-shift control alongside the existing ablations (Table 2), Qwen3-4B, average accuracy (%).

MethodMATH500AIME24AIME25AMC23MinervaOlymp.HMMT25AIME26MacroΔ vs. GRPO
GRPO92.1041.3535.3187.1143.8467.0618.8536.7752.80—
No alignment93.2746.4638.6587.8944.3968.4321.1540.9455.15+2.35
No centering95.1563.8551.3593.3646.6073.9138.6554.4864.67+11.87
Shuffled shift (new)90.2233.3630.0882.5441.9162.1720.0336.6949.63−3.17
Delta-GRPO95.5066.8855.2196.1746.2374.8736.9859.2766.39+13.59

Implementation. At every update, we draw a fresh permutation of zi,tz_{i,t} over the valid tokens of each response, before alignment and centering. Because sign(Ai)\mathrm{sign}(A_i) and zˉi\bar z_i are constant within a response, every response receives exactly the same multiset of multipliers as in the full method. All other settings follow Table 2.

Why this matters, and the relation to §5.5. By Eq. (9), the update added by Delta-GRPO is the within-response covariance between the shift and the token score gradients. The shuffle keeps every scalar statistic of the shift but makes this covariance random. Near-uniform multiplier mass is a property of the construction (multipliers lie in [1−ηk,1+ηk][1-\eta_k,1+\eta_k] with mean exactly one), so it holds for the shuffled allocation as well. What differs is where the deviations are placed, and that placement turns a 13.59-point gain into a 3.17-point loss.

Token-level case studies (Appendix E.3, Figure 8). Figure 8 visualizes mi,t−1m_{i,t}-1 token by token across four responses (two correct and two incorrect), with red/blue indicating multipliers above/below one. In the two correctly solved, positive-advantage cases, self-verification windows (e.g., "But wait, let me think" and "But wait, maybe I need to think more generally…") have mean multipliers of 1.00345 and 1.00407, versus 0.99680 and 0.99749 for the respective final-answer windows. Relative to uniform GRPO weighting, this redistributes positive credit toward self-verification reasoning and away from final-answer tokens.

Manuscript Revision. We have added the shuffled-shift row to Table 2 with discussion in §5.2, and its implementation to Appendix C.3.

AuthorsOpenReview ↗

Response to Reviewer Utxc [Part 2/3]

2. Matched Elapsed Training Time (W2)

Delta-GRPO reaches better results in fewer steps and stays ahead as training continues, and its training reward is also higher at nearly matched elapsed training times.

(1) Fewer steps, better results, sustained lead. On online MATH500 validation, Delta-GRPO is ahead from step 20 (84.33 vs. 83.90), exceeds GRPO's final step-140 accuracy at step 60 (92.03 vs. 91.80), and keeps improving to 93.98 at step 140. Checkpoints taken well before GRPO's 20.668 training hours already exceed GRPO's final checkpoint on all three benchmarks validated online:

Table R2. Online validation accuracy (%) within the GRPO wall-clock budget, Qwen3-4B.

BenchmarkGRPO, step 140 (20.668 h)Delta-GRPO, earlier checkpointΔ
MATH50091.8092.20 (step 80, 15.796 h)+0.40
AIME2441.8846.04 (step 70, 12.755 h)+4.16
AIME2533.1336.77 (step 70, 12.755 h)+3.64

(2) Training reward at nearly matched time. The outcome reward is logged at every step. At 20.668 h (GRPO, step 140) and 20.471 h (Delta-GRPO, step 96), the respective 10-step mean outcome rewards are 0.3232 and 0.3342.

Manuscript Revision. We have added the matched-time comparison to Appendix C.3 and referenced it in §5.4.

3. Optimization Regime (W3)

This setting is documented in Table 6 (inner epochs = 1) and Appendix C.1, which states that equal prompt-batch and mini-batch sizes yield one optimizer update per collected rollout batch and that the recorded clipping fractions are zero. Following your suggestion, we have moved this statement into the main text directly after Eq. (5):

In all reported experiments, each collected rollout batch produces one optimizer update. The ratio denominator is the detached current policy in this single-update regime, so the measured clipping fraction is zero.

Manuscript Revision. The statement now appears in §3.4 after Eq. (5).

4. Selection and Extension of η_max (Q1)

η_max was selected based on validation-set performance. The extended sweep confirms that 0.05 is the best evaluated setting rather than the last point tested.

Table R3. Modulation-strength sweep, Qwen3-4B, average accuracy (%).

η_maxMATH500AIME24AIME25AMC23MinervaOlymp.HMMT25AIME26Macro
0 (GRPO)92.1041.3535.3187.1143.8467.0618.8536.7752.80
0.0192.9244.5832.9287.6645.8667.8418.1238.6553.57
0.0394.5854.5842.1991.5646.6569.9927.1947.6059.29
0.0595.5066.8855.2196.1746.2374.8736.9859.2766.39
0.07 (new)95.6164.0352.1793.7344.8873.6029.6960.5264.28
0.09 (new)94.9256.2452.2590.2945.7570.5127.7056.5861.78

Macro accuracy peaks at 0.05 and declines beyond it, while 0.07 and 0.09 remain well above GRPO (+11.48 and +8.98), so 0.05 lies inside a broad effective region.

Manuscript Revision. We have extended Table 3 (§5.3) to η_max = 0.09 and stated the selection basis in §5.3.

AuthorsOpenReview ↗

Response to Reviewer Utxc [Part 3/3]

5. Per-Seed Macro Scores (Q2)

The per-seed scores, overall macro results, and question-level bootstrap intervals are shown below.

Table R4. Per-seed 8-benchmark macro accuracy (%).

ModelMethodSeed 1Seed 2Seed 3Overall ± seed SDPaired 95% bootstrap CI
Qwen3-4BGRPO53.2153.0652.1352.80 ± 0.59
Qwen3-4BDelta-GRPO66.3365.1067.7466.39 ± 1.32
Qwen3-4BPaired gain+13.12+12.04+15.61+13.59 ± 1.83[11.17, 16.09]
Qwen3-8BGRPO56.8755.1554.1855.40 ± 1.36
Qwen3-8BDelta-GRPO67.1367.9867.2467.45 ± 0.46
Qwen3-8BPaired gain+10.26+12.83+13.06+12.04 ± 1.55[10.10, 14.00]

Delta-GRPO exceeds GRPO on every seed at both scales, and the seed-level standard deviation of the paired gain is small relative to the gain itself.

Manuscript Revision. We have added the per-seed macro results to Appendix D.3 (Table 16).

6. EP-GRPO: Original vs. Shared Setting (Q3)

Our implementation follows the original EP-GRPO method and its EP hyperparameters exactly; the weak result stems from non-terminating generation. Under the shared setting, EP-GRPO policies fall into repetitive output loops and fail to stop, so responses are truncated at the length cap: at 8B, 13,472 of 16,688 endpoint samples (80.73%) reach the 16,384-token evaluation cap.

Table R5. Original vs. shared setting.

EP-GRPO originalShared setting (ours)
ModelQwen2.5-3B / 7BQwen3-4B / 8B
EP hyperparametersλ=0.1, η=0.2, 10 buckets, γ=5identical
Generation length (training / evaluation)2,048 / 2,0488,192 / 16,384
Base → EP-GRPO18.14 → 22.93 (3B); 27.11 → 30.34 (7B)38.99 → 42.10 (4B); 39.70 → 31.52 (8B)

The original paper operates with 2,048-token generation, where non-termination is cut off early; the long-reasoning setting of our benchmarks exposes it. The main comparisons rest on the matched GRPO pair and on SC-GRPO, the strongest baseline at both scales (57.49 and 59.56).

Manuscript Revision. We have added the EP-GRPO setting comparison and cap-hit rate to Appendix C.3 (Table 11).

7. Teacher Preparation vs. Student Training Cost (Q4)

The two costs are now separated beside the results.

Table R6. Cost and accuracy, Qwen3-4B (140 updates each).

MethodTeacher preparationStudent / main trainingTotalRollouts / promptMacro
GRPO—20.668 h20.668 h852.80
ExOPD20.668 h (GRPO)12.953 h33.621 h153.26
Direct-OPD20.668 h (GRPO)13.242 h33.910 h149.84
Delta-GRPO—32.168 h32.168 h866.39

Manuscript Revision. Section 5.4 summarizes total training costs; Appendix C.3 separates teacher preparation and main training costs alongside accuracy (Table 10).

8. Reward Definition in Figures 1 and 2 (Q5)

We have revised both figures to reflect the training formulation. Figure 1 distinguishes reinforcement and suppression by the sign of the group-relative advantage. Figure 2 decomposes the shaped reward into verifier correctness (±1) and the length penalty.

Manuscript Revision. Figures 1–2 are redrawn in the revised manuscript.

We hope these clarifications and additional results address your concerns. We welcome any further questions or discussion.

Reviewer_5dqNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Delta-GRPO use the log-likelihood difference between the initial model and rollout model to provide informative credit assignment to intermediate tokens, in addition to GRPO's outcome-level advantages, together with several techniques, including clip & scale, alignment, centering, etc.

Experiments with Qwen3-4B and Qwen3-8B across eight mathematics benchmarks report macro-accuracy improvements of 13.59 and 12.04 percentage points over GRPO, and 8.90 and 7.89 points over SC-GRPO.

Critical strengths
  1. A compact, well-defined intervention. The method tries to avoid separate trained teacher or process labels, but still provides meaningful credit assignment.
  2. Large and consistent reported improvements. Table 1, 2, 3, 4 provides large and consistent improvements on most of the mathematical reasoning tasks.
Critical weaknesses
  1. The evidence does not establish whether historical changes provide informative token allocation directly. The experimental results are mostly reported accuracy. Historical probability changes can reflect useful learned preferences, but can also reflect incidental correlations, changes in style, or preferences that are no longer appropriate under the current policy.
  2. Some part of the paper still need polishing.
Questions for authors
  1. Can the authors provide more detailed experiments on the evidence that historical policy changes can provide more informative credit assignment, except from the final reported task accuracy? Or can the author provide some detailed explanation (or case study) on the benefits of historical policy changes, like how useful reasoning tokens are encouraged in this formulation?
  2. How is Delta-GRPO compared with OPD from a better teacher than GRPO trained one on the same model?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 5dqN [Part 1/2]

Dear Reviewer 5dqN,

Thank you for your careful review and constructive feedback. We appreciate your characterization of the method as a compact, well-defined intervention that provides credit assignment without a separately trained teacher or process labels. Below, we address your points one by one.

1. Evidence that Historical Shifts Provide Informative Credit (W1, Q1)

The construction prevents stale preferences from overriding the current outcome, a new shuffled-shift control isolates the information carried by the token-to-history correspondence, and case studies show how reasoning tokens are encouraged.

Theoretical safeguard. The external answer verifier provides fresh correctness feedback for each rollout; together with length shaping, it determines the group-relative advantage AiA_i. The historical signal is aligned with the sign of the current advantage, gated by ∣Ai∣|A_i|, and centered within the response, producing positive multipliers within 1±ηk1\pm\eta_k (ηk≤0.05\eta_k\le0.05). The current advantage therefore retains its sign and total credit, while history determines the token-level allocation (Proposition 1, Eq. 6).

Shuffled-shift control (new). Within each response, we permute the shift values. Every response receives exactly the same set of multipliers as in the full method; only which token receives which value changes. The no-alignment variant removes the orientation by the current advantage.

Table R1. Qwen3-4B, average accuracy (%).

MethodMATH500AIME24AIME25AMC23MinervaOlymp.HMMT25AIME26MacroΔ vs. GRPO
GRPO92.1041.3535.3187.1143.8467.0618.8536.7752.80—
No alignment93.2746.4638.6587.8944.3968.4321.1540.9455.15+2.35
Shuffled shift (new)90.2233.3630.0882.5441.9162.1720.0336.6949.63−3.17
Delta-GRPO95.5066.8855.2196.1746.2374.8736.9859.2766.39+13.59

If the shift reflected only incidental correlations or a global change in style, any assignment with the same multiplier set would perform similarly. Instead, removing the correspondence drops macro accuracy from 66.39 to 49.63, below GRPO, and removing the orientation by the current outcome keeps only 2.35 of the 13.59-point gain.

AuthorsOpenReview ↗

Response to Reviewer 5dqN [Part 2/2]

Token-level case studies (Appendix E.3, Figure 8). Figure 8 visualizes mi,t−1m_{i,t}-1 token by token across four responses (two correct and two incorrect), with red/blue indicating multipliers above/below one. For the two correctly solved, positive-advantage cases, Table R2 compares self-verification windows (e.g., "But wait, let me think" and "But wait, maybe I need to think more generally…") with their final-answer windows.

Table R2. Mean token multiplier in two correctly solved responses (positive advantage).

ProblemSelf-check windowSelf-checkFinal answer
Circular seating"we need to adjust for rotational symmetry. But wait, let me think"1.003450.99680
Grid counting"But wait, maybe I need to think more generally about all possible squares"1.004070.99749

Relative to uniform GRPO weighting, the positive-advantage responses assign more credit to self-verification reasoning (mean multipliers 1.00345 and 1.00407) and less to answer-emitting tokens (0.99680 and 0.99749).

Manuscript Revision. We have added the shuffled-shift row to Table 2 (§5.2).

2. Comparison with OPD from a Stronger Teacher (Q2)

We used Delta-GRPO itself, the strongest same-size model available (66.39 vs. 52.80 macro for GRPO), as the OPD teacher. We kept the ExOPD objective, student initialization, training data, rollout count, update count (140) and learning rate fixed, and replaced only the frozen GRPO teacher.

Table R3. OPD with a stronger teacher, Qwen3-4B, average accuracy (%).

MethodMATH500AIME24AIME25AMC23MinervaOlymp.HMMT25AIME26Macro
OPD (GRPO teacher; ExOPD)91.4242.7134.0687.9743.8066.3919.0640.6253.26
OPD (Delta-GRPO teacher, new)93.4559.1140.2789.7545.2268.0329.5042.6858.50
Delta-GRPO95.5066.8855.2196.1746.2374.8736.9859.2766.39

The stronger teacher raises OPD from 53.26 to 58.50, and Delta-GRPO remains higher on all eight benchmarks and by 7.89 points macro. OPD from a substantially larger external teacher could exceed these results; that setting, however, requires access to a stronger model and a separate distillation stage. Delta-GRPO's advantage is that it obtains dense token-level guidance from the learner's own training history, with no external teacher.

Manuscript Revision. We have added the stronger-teacher OPD comparison and its configuration to Appendix C.3 (Table 8).

3. Presentation (W2)

We have revised the presentation in the updated manuscript:

LocationRevision
Figures 1–2Figure 1 distinguishes reinforcement and suppression by the sign of the group-relative advantage; Figure 2 shows verifier correctness (±1) and the length penalty in the shaped reward
§3Notation and wording clarified
§3.4One-update-per-batch setting stated directly after Eq. (5)

We will continue to refine the writing and presentation in the final version.

We hope these clarifications and additional results address your concerns. We welcome any further questions or discussion.

Reviewer_5dqNOpenReview ↗

Thanks for the Response. Maintain My Score Currently.

Thank you for the detailed response and the new experiments. The case study and the shuffled-shift control are the kind of evidence I asked for in Q1, and the stronger-teacher comparison addresses the form of Q2. I am maintaining my score for now. To make the results easier to interpret and replicate, could you clarify the following? I believe none of these needs new training.

  1. Hardware. Which NPU model, how many devices, and how much memory per device were used for Qwen3-4B, 8B and 32B? What was the wall-clock time of the 32B runs?

  2. Qwen3-32B setup. How many updates and training seeds were used? What is the detailed training and evaluation setting for 32B model?

  3. Qwen3-32B evaluation. The 32B result is interesting. Could you report the remaining five benchmarks used for 4B and 8B (AMC23, MinervaMath, OlympiadBench, HMMT25, AIME26), the untrained Qwen3-32B as a reference row?

Thank you.

Reviewer_65niOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Delta-GRPO, a token-level credit assignment method that integrates policy-shift guidance into GRPO. It uses accumulated changes in token probabilities to redistribute outcome advantages while preserving their mean and sign. Experiments on two Qwen3 models across eight mathematics benchmarks demonstrate substantial improvements over existing RL and OPD baselines.

Critical strengths
  1. The paper introduces a simple token-level credit redistribution mechanism that integrates historical policy shifts into GRPO without requiring an external teacher.

  2. Delta-GRPO achieves substantial and consistent improvements across eight mathematics benchmarks and two model sizes, supported by component ablations.

Critical weaknesses
  1. The use of historical policy shifts for token-level credit assignment lacks sufficient justification, as these shifts may reinforce previously learned errors. The theoretical analysis does not establish improved optimization.

  2. Delta-GRPO incurs 55.6% higher per-step training costs, while the baseline comparisons are not fully compute-matched, weakening the empirical evidence.

Questions for authors
  1. What evidence shows that historical policy shifts reflect meaningful token contributions rather than reinforcing previously learned errors?

  2. How does the theoretical analysis support improved optimization beyond preserving the mean and sign of advantages?

  3. Do the performance gains persist when all baselines are compared under matched computational budgets?

  4. Could the longer responses and substantially lower token entropy explain part of the reported improvements?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 65ni [Part 1/3]

Dear Reviewer 65ni,

Thank you for your careful review and constructive feedback. We appreciate your recognition that the method redistributes token-level credit within GRPO without an external teacher. Below, we address your concerns point by point.

1. Historical Shifts and Previously Learned Errors (W1, Q1)

The external answer verifier supplies fresh correctness feedback on current rollouts; historical shifts redistribute the resulting group-relative advantage across tokens. The shuffled-shift control tests this token-to-history correspondence, and Figure 8 visualizes the token-level allocation.

Theoretical safeguard. The external answer verifier evaluates current rollouts, and its correctness feedback, combined with length shaping, determines the group-relative advantage AiA_i. Historical shifts redistribute this advantage through positive multipliers mi,t∈[1−ηk,1+ηk]m_{i,t}\in[1-\eta_k,1+\eta_k] (ηk≤0.05\eta_k\le0.05). For Ai<0A_i<0, every token retains a negative advantage (Aimi,t<0A_i m_{i,t}<0), and the response's total penalty remains TiAiT_iA_i (Proposition 1, Eq. 6). Fresh outcome feedback therefore guides the correction of previously learned preferences, while history determines the token-level allocation.

Shuffled-shift control (new). Within each response, we permute the shift values. This keeps their distribution, centering and alignment, and changes only which token receives which value. The no-alignment variant tests the opposite treatment of past reinforcement: on negative advantages it penalizes historically reinforced tokens more.

Table R1. Qwen3-4B, average accuracy (%).

MethodMATH500AIME24AIME25AMC23MinervaOlymp.HMMT25AIME26MacroΔ vs. GRPO
GRPO92.1041.3535.3187.1143.8467.0618.8536.7752.80—
No alignment93.2746.4638.6587.8944.3968.4321.1540.9455.15+2.35
Shuffled shift (new)90.2233.3630.0882.5441.9162.1720.0336.6949.63−3.17
Delta-GRPO95.5066.8855.2196.1746.2374.8736.9859.2766.39+13.59

With identical multiplier sets, removing the token-to-history correspondence drops macro accuracy from 66.39 to 49.63, below GRPO. Penalizing historically reinforced tokens more keeps only 2.35 of the 13.59-point gain. The gain therefore comes from which token carries which historical value, oriented by the current outcome.

Token-level case studies (Appendix E.3, Figure 8). Figure 8 visualizes mi,t−1m_{i,t}-1 token by token across four responses (two correct and two incorrect), with red/blue indicating multipliers above/below one. For the two correctly solved, positive-advantage cases, Table R2 compares self-verification windows (e.g., "But wait, let me think" and "But wait, maybe I need to think more generally…") with their final-answer windows.

Table R2. Mean token multiplier in two correctly solved responses.

ProblemSelf-check windowSelf-checkFinal answer
Circular seating"we need to adjust for rotational symmetry. But wait, let me think"1.003450.99680
Grid counting"But wait, maybe I need to think more generally about all possible squares"1.004070.99749

Relative to uniform GRPO weighting, the positive-advantage responses assign more credit to self-verification reasoning (mean multipliers 1.00345 and 1.00407) and less to answer-emitting tokens (0.99680 and 0.99749).

Manuscript Revision. We have added the shuffled-shift row to Table 2 (§5.2) and its implementation to Appendix C.3.

AuthorsOpenReview ↗

Response to Reviewer 65ni [Part 2/3]

2. How the Analysis Supports Improved Optimization (W1, Q2)

Beyond mean and sign preservation, the analysis characterizes the direction of the added update: under a local linear approximation, its projection onto the parameter change accumulated by earlier RL updates is non-negative; the update acts only along directions in which the tokens of a response differ and is gated by the current outcome. Whether this structured bias helps is then tested directly by the ablations.

Projection onto the accumulated direction. Let d=θold−θ0d=\theta_{\rm old}-\theta_0, and let gi,t=∇θlog⁡πθ(yi,t∣si,t)g_{i,t}=\nabla_\theta\log\pi_\theta(y_{i,t}\mid s_{i,t}) be evaluated at θold\theta_{\rm old}. By Eq. (9), response ii adds δgi=ηk∣Ai∣2Ti∑t(zi,t−zˉi)(gi,t−gˉi)\delta g_i=\frac{\eta_k|A_i|}{2T_i}\sum_t(z_{i,t}-\bar z_i)(g_{i,t}-\bar g_i) to the GRPO gradient, so exactly

⟨δgi,d⟩=ηk∣Ai∣2,Covi(zi,t,,gi,t⊤d)\langle\delta g_i,d\rangle=\frac{\eta_k|A_i|}{2}\\,\mathrm{Cov}_i\big(z_{i,t},\\,g_{i,t}^\top d\big)

where Covi\mathrm{Cov}_i is the empirical covariance over the tokens of the response. Under the local linear approximation Δi,t≈gi,t⊤d\Delta_{i,t}\approx g_{i,t}^\top d, zi,tz_{i,t} is a non-decreasing function of gi,t⊤dg_{i,t}^\top d (clipping and rescaling are monotone), so this covariance is non-negative however many tokens are clipped. When no token of the response is clipped, the term takes the closed form

δgi≈ηk∣Ai∣2ck C^i d,C^i=1Ti∑t(gi,t−gˉi)(gi,t−gˉi)⊤⪰0.\delta g_i\approx\frac{\eta_k|A_i|}{2c_k}\,\hat C_i\,d,\qquad \hat C_i=\frac1{T_i}\sum_t(g_{i,t}-\bar g_i)(g_{i,t}-\bar g_i)^\top\succeq 0.

Because the objective averages over all valid tokens, the batch-level term is ∑iTiNδgi\sum_i\frac{T_i}{N}\delta g_i, whose projection onto dd is likewise non-negative under the same approximation.

Interpretation. The added term is a bounded bias that continues the learner's accumulated change, restricted to directions that distinguish the tokens of a response (C^i\hat C_i), and scaled by ∣Ai∣|A_i| and ηk\eta_k, so it vanishes when the current reward carries no signal and at the start and end of training (Eq. 8). The verifier-grounded GRPO advantage sets the response-level direction, while the historical term redistributes credit across its tokens (Proposition 1).

Empirical test. The analysis attributes any effect to the covariance between zz and the token score gradients. The shuffled-shift control keeps every scalar statistic of zz but removes this covariance in expectation; macro accuracy falls to 49.63. Under the same approximation, removing alignment reverses the projection on negative-advantage responses; macro accuracy falls to 55.15. With the full construction, Delta-GRPO reaches GRPO's final step-140 MATH500 validation accuracy at step 60 (92.03 vs. 91.80) and ends 13.59 points higher in macro accuracy. These results are consistent with the analysis and indicate that this structured bias benefits optimization in practice.

Manuscript Revision. We have added this analysis to Appendix B.2.

AuthorsOpenReview ↗

Response to Reviewer 65ni [Part 3/3]

3. Matched Computational Budgets (W2, Q3)

Delta-GRPO reaches better results in fewer steps and stays ahead as training continues, and its training reward is also higher at nearly matched elapsed training times.

(1) Fewer steps, better results, sustained lead. On online MATH500 validation, Delta-GRPO is ahead from step 20 (84.33 vs. 83.90), exceeds GRPO's final step-140 accuracy at step 60 (92.03 vs. 91.80), and keeps improving to 93.98 at step 140. On all three benchmarks validated online, Delta-GRPO checkpoints taken well before GRPO's 20.668 training hours already exceed GRPO's final checkpoint:

Table R3. Online validation accuracy (%) within the GRPO wall-clock budget, Qwen3-4B.

BenchmarkGRPO, step 140 (20.668 h)Delta-GRPO, earlier checkpointΔ
MATH50091.8092.20 (step 80, 15.796 h)+0.40
AIME2441.8846.04 (step 70, 12.755 h)+4.16
AIME2533.1336.77 (step 70, 12.755 h)+3.64

(2) Training reward at nearly matched time. The outcome reward is logged at every step. At 20.668 h (GRPO, step 140) and 20.471 h (Delta-GRPO, step 96), the respective 10-step mean outcome rewards are 0.3232 and 0.3342.

OPD baselines. ExOPD and Direct-OPD require a full GRPO teacher run before student training.

Table R4. Total training cost, Qwen3-4B (140 updates each).

MethodTeacher preparationStudent / main trainingTotalMacro
GRPO—20.668 h20.668 h52.80
ExOPD20.668 h12.953 h33.621 h53.26
Direct-OPD20.668 h13.242 h33.910 h49.84
Delta-GRPO—32.168 h32.168 h66.39

Manuscript Revision. We have added the matched-time validation and total training-cost comparisons to Appendix C.3 (Tables 9–10).

4. Response Length and Token Entropy (Q4)

Longer reasoning explains part of the gain, which is expected for RL on mathematics; DAPO (Yu et al., 2025) likewise reports that growing response length accompanies improved reasoning. The gain, however, persists at equal output-token cost, and lower entropy does not reduce solution diversity.

Length. At the expected output-token cost of 32 GRPO samples (Table 15), Delta-GRPO still covers more AIME questions: 88.69 vs. 73.33 on AIME24 and 84.32 vs. 70.00 on AIME25 (+15.35 / +14.32).

Entropy. If lower entropy reflected collapsed exploration, Delta-GRPO would gain mainly at pass@1, and the gain would shrink toward zero as kk grows. Table R5 shows otherwise.

Table R5. pass@1 vs. pass@k, Qwen3-4B (%). pass@1 equals the average accuracy in Table 1; kk is the full sample bank.

Benchmark (kk)GRPO @1Delta @1Gain @1GRPO @kkDelta @kkGain @kk
MATH500 (8)92.1095.50+3.4097.6099.00+1.40
AIME24 (32)41.3566.88+25.5373.3390.00+16.67
AIME25 (32)35.3155.21+19.9070.0090.00+20.00
AMC23 (32)87.1196.17+9.0697.50100.00+2.50
Minerva (8)43.8446.23+2.3955.1555.88+0.73
Olymp. (8)67.0674.87+7.8178.9385.46+6.53
HMMT25 (32)18.8536.98+18.1343.3370.00+26.67
AIME26 (32)36.7759.27+22.5063.3386.67+23.34

The gain remains positive at large kk on all eight benchmarks, and on AIME25, HMMT25 and AIME26 it is larger at pass@32 than at pass@1. Smaller pass@kk gains appear mainly where GRPO is already near saturation (MATH500 97.60, AMC23 97.50). Lower next-token entropy on training states thus coexists with broader solution coverage.

Manuscript Revision. We have added the pass@1 versus pass@k comparison to Appendix D.1 (Table 14).

We hope these clarifications and additional results address your concerns. We welcome any further questions or discussion.

AuthorsOpenReview ↗

A Kind Reminder for Discussion

Dear Reviewer 65ni,

Thank you again for your time and constructive comments on our work.

We have submitted our detailed response, which we hope will address your initial concerns. As the author–reviewer discussion period is coming to an end, we respectfully ask if you could review our updates at your convenience.

If you find that our response has adequately resolved your concerns, we would be deeply grateful if you might consider reflecting this in your final evaluation. We would be happy to provide any further clarification if needed.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all four reviewers for their careful evaluations and constructive suggestions. Our work proposes Delta-GRPO, which uses the learner's accumulated policy shift to guide token-level credit allocation within GRPO without requiring a separate teacher or distillation stage. We appreciate the reviewers' recognition of the method and its consistent empirical improvements. In response to their feedback, we have provided additional experiments, theoretical analysis, and implementation clarifications, summarized below.

1. Historical Policy Shifts and Token-Level Credit Assignment (Reviewers 65ni, 5dqN, Utxc, qbaV)

We add a shuffled-shift control that preserves the token-weight distribution while breaking the token-to-history correspondence. The 4B macro accuracy drops from 66.39% to 49.63%, below GRPO's 52.80%. Together with token-level case studies and further optimization analysis, this provides direct evidence for the role of historical shifts.

2. Relation to Existing Methods and Stronger-Teacher OPD (Reviewers 5dqN, qbaV)

We clarify the distinctions from Direct-OPD, OPD², EP-GRPO, and SC-GRPO. We also compare OPD with a stronger Delta-GRPO teacher: its macro accuracy increases from 53.26% to 58.50%, while Delta-GRPO itself reaches 66.39%.

3. Training Cost and Matched-Time Comparisons (Reviewers 65ni, Utxc, qbaV)

We report detailed training costs, including separate teacher-preparation and student-training costs for OPD baselines. Matched-time evidence on three online 4B validation benchmarks shows that earlier Delta-GRPO checkpoints already outperform GRPO's final checkpoint within a shorter wall-clock budget; we also report training rewards at nearly matched elapsed times.

4. Response Length, Exploration, and Solution Coverage (Reviewers 65ni, qbaV)

We add pass@1 versus pass@k and matched output-token-budget analyses. The results show that Delta-GRPO's gains persist across sampling budgets and are not solely explained by longer responses, while lower token entropy coexists with higher empirical solution coverage.

5. Robustness, Scaling, and Cross-Domain Generalization (Reviewers Utxc, qbaV, 5dqN)

We report multi-seed results for 4B/8B, extend the eight-benchmark evaluation to Qwen3-32B, and add five code benchmarks. Delta-GRPO improves eight-benchmark macro accuracy over GRPO by 13.59, 12.04, and 11.91 percentage points at 4B, 8B, and 32B, respectively.

6. Experimental Settings, Baseline Diagnostics, and Presentation (Reviewers Utxc, qbaV, 5dqN)

We expand the η_max sweep, examine EP-GRPO's behavior under the shared long-response setting, clarify the one-update-per-rollout-batch regime and experimental configurations, and improve the reward illustrations, method descriptions, and reproducibility details.

We believe these additions have substantially addressed the reviewers' main concerns. The corresponding experimental results, analyses, and clarifications have been incorporated into the revised manuscript, with changes highlighted in blue. We sincerely thank all reviewers again for their time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.