- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper identifies the limitation of vanilla GRPO where the same advantage of a response is assigned to all tokens regardless of their contribution, and proposes Direction-Aware Token Credit Assignment (DATC). DATC utilizes a straight-through (ST) surrogate of the reward model gradient w.r.t. token logits, and computes their inner products with the standard GRPO token-probability update directions to adaptively reweight token-level advantages. Experiments on Qwen3-14B and Llama-3.1-8B demonstrate faster reward improvement over naive GRPO on different benchmarks(RM-Reward, AlpacaEval, Arena-Hard, MT-Bench) with modest computational overhead.
Critical strengths
- The authors provided a theoretical justification that introducing such adaptive advantage allocation to each tokens guarantees somewhat nonnegative reward gain against naive GRPO, beyond naive empirical design choice.
- DATC assigns higher weights to informative or corrective tokens in high-reward responses and erroneous tokens in low-reward responses to enable more targeted reinforcement and suppression, which can provide an explainability of the model behavior to improve the reward.
Critical weaknesses
- Reward model has to be prepared such that it has the same tokenization space with the actor LLM.
- The gradient signal of the reward function is essential to apply DATC. This not only limits the applicability of the proposed method toward many black-box rewards(e.g. criterion/rule-based rewards), this makes the comparison of DATC between naive GRPO unfair since GRPO only recieves the zeroth-order reward signal.
Questions for authors
- If one can access to the reward model gradient, isn't it natural to naively perform reward gradient ascent(+KL divergence with the reference policy) into the model? Could the authors provide the performance of this approach?
- Beyond qualitative examples such as Figure 4, could the authors provide quantitative evidence that tokens more relevant to the reward receive more favorable dot-product weights?
Thank you for your suggestions; we have redrawn Figure 2 and reorganized the Method section to present the computation more clearly, while retaining Figure 1 as a teaser to introduce the problem, build intuition, and engage readers. We hope these revisions address your concerns and that you will consider these improvements in your final assessment.