acceptodds
Under review as a conference paper at ICLR 2027

Direction-Aware Token Credit Assignment with Reward-Model Gradients

Abstract

GRPO shares a response-level advantage across tokens, leaving unresolved how strongly each token should contribute to learning. Updates induced by the shared advantage reinforce or suppress the sampled token at each position, but these updates do not always point toward increasing reward at the token level. This motivates a **work-based view of credit**: just as work depends on the component of force along a displacement, token credit should reflect the local reward change along the policy update direction. We introduce **Direction-Aware Token Credit Assignment (DATC)**, which maps frozen reward-model gradients into policy logit space and allocates weights by evaluating the alignment between the reward direction and GRPO's signed sampled-action update direction, while preserving the sign and mean of the response-level advantage. DATC outperforms GRPO in all four settings across two model families. Our code will be released at https://anonymous.4open.science/r/DATC-0161.

Primary area:
reinforcement learning
Keywords:
Token-level Credit Assignment, Group Relative Policy Optimization, Reward Modeling, Policy Gradients, LLM Post-Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cAUnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper identifies the limitation of vanilla GRPO where the same advantage of a response is assigned to all tokens regardless of their contribution, and proposes Direction-Aware Token Credit Assignment (DATC). DATC utilizes a straight-through (ST) surrogate of the reward model gradient w.r.t. token logits, and computes their inner products with the standard GRPO token-probability update directions to adaptively reweight token-level advantages. Experiments on Qwen3-14B and Llama-3.1-8B demonstrate faster reward improvement over naive GRPO on different benchmarks(RM-Reward, AlpacaEval, Arena-Hard, MT-Bench) with modest computational overhead.

Critical strengths
  • The authors provided a theoretical justification that introducing such adaptive advantage allocation to each tokens guarantees somewhat nonnegative reward gain against naive GRPO, beyond naive empirical design choice.
  • DATC assigns higher weights to informative or corrective tokens in high-reward responses and erroneous tokens in low-reward responses to enable more targeted reinforcement and suppression, which can provide an explainability of the model behavior to improve the reward.
Critical weaknesses
  • Reward model has to be prepared such that it has the same tokenization space with the actor LLM.
  • The gradient signal of the reward function is essential to apply DATC. This not only limits the applicability of the proposed method toward many black-box rewards(e.g. criterion/rule-based rewards), this makes the comparison of DATC between naive GRPO unfair since GRPO only recieves the zeroth-order reward signal.
Questions for authors
  • If one can access to the reward model gradient, isn't it natural to naively perform reward gradient ascent(+KL divergence with the reference policy) into the model? Could the authors provide the performance of this approach?
  • Beyond qualitative examples such as Figure 4, could the authors provide quantitative evidence that tokens more relevant to the reward receive more favorable dot-product weights?
Reviewer_aonrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to utilize reward model gradients as a signal to improve credit assignment in GRPO. The proposed pipeline uses a straight-through estimator to propagate the reward model gradients back to the LLM logits, and use this information to reweigh the GRPO advantage weights. The experimental results show strong gains, and the interpretability analysis is also convincing.

Critical strengths

Using reward model gradients to get more information out of the reward model is interesting, the results are strong, and the interpretability figure is also very intersting.

Critical weaknesses

I have some issues with the exposition of the idea and the writing of paper:

  • figures 1 and 2 are not really informative, I appreciate authors' efforts to make them aesthetic, but they carry almost no information. i suggest the authors create figures that actually show the flow of each token's computation to improve exposition.
  • It it quite hard to understand sections 3.2 and 3.3, which are the main ingredients of deriving the weighting mechanism. I encourage the authors to better motivate those sections.
  • The method has too many hyperparameters and tricks (e.g., lambda). It might have been better to remove some of those since they still outperform GRPO and tuning them only adds overhead.
Questions for authors
  • Although I understand the motivation of using more of the reward model gradients, the current straight through is rather counterintuitive. Because in principle we can add any kind of straight through gradients in the nework. what is the motivation behind doing it at the encoder level? and on the logits?
2 comments

Thank you for your suggestions; we have redrawn Figure 2 and reorganized the Method section to present the computation more clearly, while retaining Figure 1 as a teaser to introduce the problem, build intuition, and engage readers. We hope these revisions address your concerns and that you will consider these improvements in your final assessment.

AuthorsOpenReview ↗

Follow-up on Our Response

Thank you again for your helpful suggestions. Please let us know if there is anything we could clarify further.

Reviewer_bh6AOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
3
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This work analyzes token-level credit assignment in RL training, aiming to improve the standard approach where the reward is applied uniformly to all tokens. The proposed method DATC weights tokens by alignment between its GRPO gradient and reward gradient, to emphasize tokens whose update direction is positively aligned with the reward-increasing direction. It first uses the step-through to estimate reward gradient on every token, instead of just the sampled token. Then redistributes response-level advantages to each token in the response, while preserving their sign and mean.

Critical strengths
  1. Addresses the important question of credit assignment in RL training, where reward is sparse and applied uniformly to all tokens
  2. The formulation and derived connections / interpretations are interesting
  3. The presentation is clean and well-organized
Critical weaknesses

The main concern is the limited evaluation against baselines. I would raise my score given sufficient comparisons of related methods:

  1. Baselines & Evaluation: Main experiments only compared with vanilla GRPO, and related methods (P2T, RED) are evaluated only in ablation on Qwen3-14B-Base.

    • (a) Additional comparisons with other gradient-based reweighting methods such as [1, 2], or approaches using policy-intrinsic signals without relying on reward gradients like [3] would better establish the value of DATC’s reward-model gradient information.

    • (b) For ablation study, a more direct control would be to reverse the assignment strategy, by upweighting tokens with negative alignment between reward gradient and GRPO gradient, especially under different λ\lambda

  2. Hyperparameter: The method seems fairly sensitive to the choice of λ\lambda. It should be helpful to (i) extend the analysis of how λ\lambda affects weight allocation in practice, and (ii) provide general guidance / intuition for selecting this hyperparameter

References

[1] Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning (arXiv:2601.07408)
[2] DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards (arXiv:2605.21467)
[3] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (arXiv:2506.01939)

Questions for authors
  1. Is DATC more susceptible to reward hacking, due to its reliance on reward gradient?
  2. Sec 3.2 shows that the reward gradient can be decomposed into advantages over the entire vocabulary, but DATC reduces this information to a scalar weight for the sampled token, leaving the relative updates of other tokens determined through softmax. Are there theoretical / empirical reasons for the scalar reweighting over directly incorporating a full vocabulary-wide reward gradient direction?
  3. It would be helpful to better understand what DATC does through additional token analysis: (i) ratio of tokens down/upweighted, (ii) general patterns in those tokens through classifer (e.g., downweighted tokens may be less informative prose), beyond the 4 examples in Fig.4
6 comments
AuthorsOpenReview ↗

Response to Reviewer bh6A (1/2)

Thank you for your positive assessment and specific suggestions. Following your comments, we have added a comparison with DelTA and a reversed-weight assignment ablation to Table 2 of the revised manuscript.

Additional baseline and reversed-weight ablation. We conducted the new experiments on Qwen3-14B-Base using the same training initialization and policy optimization settings as the original experiments:

MethodRM-Reward ↑AlpacaEval ↑Arena-Hard ↑MT-Bench ↑
GRPO10.8737.39%28.49%7.70
DelTA12.7145.86%34.34%7.92
DATC13.3549.57%39.31%7.99
Reversed weights9.2333.60%25.44%7.41

DATC achieves higher scores than DelTA on all four metrics, improving AlpacaEval and Arena-Hard by 3.71 and 4.97 percentage points, respectively.

Given the time available during the rebuttal period, we prioritized the DelTA comparison and your suggested reversed-weight ablation. We selected DelTA because it also studies token-level credit assignment and reports higher scores than the high-entropy token method from Beyond the 80/20 Rule in direct comparisons on mathematical reasoning tasks. We will also conduct experiments with the high-entropy token method from Beyond the 80/20 Rule in future work.

The reversed-weight ablation preserves each response’s original set of weights and reverses their assignment according to the signed local utility AdtA d_t, assigning larger weights to positions with lower predicted update utility. This preserves the mean, range, and concentration of the weights while changing their assignment direction. Reversing the assignment reduces AlpacaEval and Arena-Hard by 15.97 and 13.87 percentage points relative to full DATC, with all four metrics falling below GRPO. These results directly support the importance of the assignment direction.

The role and choice of λ. First, λ controls the weight range and the differences in update strength across positions: weights lie within [1/λ,λ][1/\lambda,\lambda] and retain a mean of 1. Smaller λ values produce more uniform allocations, while larger values allow greater differences across positions. The temperature is determined automatically to satisfy this constraint.

Second, Table 2 shows that λ = 2, 4, and 8 all outperform GRPO on all four metrics for Qwen3-14B-Base, with λ = 4 performing best. We also use λ = 4 consistently across all four main experimental settings. We therefore recommend λ = 4 as a default starting point, with smaller values offering more conservative adjustments to update strength.

Reward hacking. First, AlpacaEval, Arena-Hard, MT-Bench, and human evaluation assess responses independently of the training reward model’s scores. DATC improves across these evaluations: on Qwen3-14B-Base, it improves AlpacaEval and Arena-Hard over GRPO by 12.18 and 10.82 percentage points, respectively, and wins 78 of 120 prompts in the four-method human comparison. These results support improvements in response quality.

Second, DATC uses λ to bound token weights within [1/λ,λ][1/\lambda,\lambda], or [0.25,4][0.25,4] in the main experiments, limiting the amplification of updates at individual positions. It also preserves the sign and mean of the response-level advantage and retains KL regularization against the reference policy. Together, these constraints limit how strongly policy updates can follow local reward gradients.

Scalar weights versus full-vocabulary gradients. First, DATC allocates update strength across positions along GRPO’s update directions. At position tt, the first-order local reward gain is proportional to Awt⟨gt,kt⟩A w_t\langle g_t,k_t\rangle, so the inner product provides the directional information needed for this allocation.

Second, the scalar weight is computed using full-vocabulary reward gradients and policy probabilities, aggregating this information into position-level credit while preserving the role of response-level scores in determining reinforcement or suppression. This allows DATC to use full-vocabulary information while explicitly constraining the influence of local surrogate gradients. We use scalar weights because DATC adjusts how strongly each position contributes to the GRPO update, while retaining its sampled-action update direction.

AuthorsOpenReview ↗

Response to Reviewer bh6A (2/2)

We continue our response below with the quantitative token-weight analysis requested in your final question.

Quantitative analysis of token weights. We have added the following analyses to address both parts of your question.

(i) Proportions of downweighted and upweighted tokens. We analyzed 16,000 training responses containing 7,750,367 valid tokens from the Qwen3-14B-Base run with λ = 4. The table reports token proportions and the mean weight change within each group relative to the uniform baseline w=1w=1.

Response groupDownweighted (w<1w<1)Unchanged (w=1w=1)Upweighted (w>1w>1)Mean weight change among downweighted tokensMean weight change among upweighted tokens
All responses64.68%7.76%27.57%−3.43%+8.05%
Positive advantage (A>0A>0)65.53%7.84%26.63%−3.40%+8.37%
Negative advantage (A<0A<0)63.83%7.52%28.65%−3.47%+7.73%

These results show that DATC reduces update magnitudes at more positions and concentrates update weights on fewer positions.

(ii) Semantic patterns identified by a classifier. We used Astra to semantically classify tokens in 100 responses. Tokens serving as generic introductions, pleasantries, and framing account for 9.47% of the lowest-weight 10% of tokens, 1.46 times their baseline proportion of 6.51%. This indicates that, in this sample, generic text carrying less direct task information is more likely to receive lower weights from DATC.

AuthorsOpenReview ↗

Follow-up on Our Response

Thank you again for your feedback. We would appreciate your thoughts on whether our responses address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.