- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies token credit assignment in GRPO/DAPO and proposes GCPO. The method uses the KL divergence between predictive distributions under positive and negative conditioning to construct token weights, applies rank-based normalization, and uses the normalized weights to scale the per-sample advantage. Experiments on text-to-image generation, multimodal reasoning, and mathematical reasoning show some improvements over the evaluated baselines.
Critical strengths
- The proposed method is simple, does not require training an additional value model or token reward model, and shows some improvements in text-to-image generation and reasoning tasks.
Critical weaknesses
- The credit assignment claim lacks direct support. The difference between predictive distributions under positive and negative prompts reflects how sensitive predictions are to the prompt conditions, but does not directly measure each token's contribution to the final reward. Since the weights are non-negative, token advantages within the same trajectory cannot have opposite signs. Correct and incorrect steps may therefore be rewarded or penalized together, which does not establish more accurate credit assignment.
- The interpretation of some experimental results overstates the contribution of token weighting. The Counting score increases from 0.59 for the base model to 0.84 for GCPO, but the reproduced GRPO baseline already reaches 0.81. The gain over this baseline is 0.03, so the entire improvement cannot be attributed to the added token weighting mechanism.
Questions for authors
- In Table 8, the reported share of reasoning tokens increases from 1.9% to 18.1%. Are all columns computed using the same set of rollouts and the same aggregation method? If the final rank-based weights are directly aggregated and their mean is approximately 0.5, with a maximum of at most 1, how can this degree of amplification be explained? Please specify the exact quantity being measured and how it is computed.
- Is the probability ratio used for training computed from the original conditional distribution or the actual sampling distribution after CFG? If the two distributions differ, how is the mismatch between the behavior policy and the target policy handled? Do GRPO and GCPO use the same implementation?
Q1
We use fixed rollout from the base checkpoint, we report tokens' weighting contribution, defined as
sums of weight of all tokenssums of weight of tokens of a particular class. Note this value is not “average weight”, but instead the “ratio” of total contribution. For example, given two token class A,B, and a 3-token sequence whose token classes are [A,B,B],one possible weight under rank normalization is [0.0,0.5,1.0]. Another is [1.0,0.5,0.0]. Comparing the two setup, the contribution ratio for class a jumps from 0% to 66.6%. Hence, increases from 1.9% to 18.1% from Table 8 is reasonable. We also note that the uniform baseline would be have a weight of [1,1,1], and class A's contribution is 33.3% under this scheme, which is just the token frequency.
Q2
We use original distribution during training and use CFG only for rollout because including CFG in RL training leads to instability. Qwen-Image-RL [1] reported similar issues
W1 Justification
We agree that correct and incorrect steps may be rewarded or penalized together. However, this limitation does not invalidate the underlying credit assignment. Existing token-level credit assignment methods, such as [2] and VPPO, operate under the same constraint. Moreover, vanilla GRPO has demonstrated that uniformly propagating sample-level rewards to all tokens can already yield performance improvements. Rather than distinguishing correct from incorrect tokens, token-level credit assignment methods investigate where the learning signal should be concentrated. For example, VPPO emphasizes visually dependent tokens, [2] focuses on high-entropy tokens, and GCPO identifies tokens that are sensitive to contrastive signals.
As clarified in our response to Q1, we report the relative contributions to the total learning weight under different weighting paradigms:
The metrics are computed using base model's rollouts.
We bold all numbers that are at least 150% of the uniform baseline, indicating that they are heavily upweighted.
Overall, compared with the baseline frequency (uniform), GCPO amplifies the learning signal primarily for math_symbolic, math_semantic, and reasoning tokens.
Comparison to Entropy: Both methods emphasize reasoning connectives such as “however” and “finally.” These tokens have already been shown in [1] to be high-entropy “forking tokens” that significantly affect learning. However, GCPO’s amplification is smoother (1.9% → 18.1% vs. 1.9% → 45.5%), and it also emphasizes low-entropy symbolic math tokens, which we argue can be equally important.
Comparison to VPPO: GCPO follows a completely different distribution from VPPO. VPPO does not heavily emphasize reasoning, math_symbolic, or math_semantic tokens. Its weight is concentrated on “content_words.” Through qualitative examination, we find that these tokens are mostly OCR text describing the image.
Importantly, the improvements do not arise from explicitly identifying correct and incorrect tokens. Rather, they arise from concentrating learning signals on task and reasoning-relevant tokens. Updating model predictions at these positions can have a greater impact on task performance than updating probabilities associated with fillers, punctuation, and other less informative tokens.
Regarding the assignment of negative credit to incorrect reasoning steps, we agree that this is a promising direction. Such fine-grained feedback is typically provided by process reward models (PRMs), which evaluate individual reasoning steps using partial reasoning traces. However, this falls into a complementary line of research and is outside the scope of our work. Our focus is on the setting where only sample-level rewards are available, without additional step-level supervision.
W2 Entire improvement cannot be attributed to the added token weighting mechanism.
We have updated the relavent section (4.1) with a more accurate analysis on the per category improvements. The section now focuses on color attribution category, which benefits from region-aware GCPO significantly.
[1] Qwen-Image-2.0-RL Technical Report
[2] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning