acceptodds
Withdrawn from ICLR 2027

Guidance-Contrastive Token Credit Assignment for Discrete Policy Optimization.

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Group-advantage-based reinforcement learning methods, such as GRPO and GSPO, have demonstrated strong performance across diverse domains, including mathematical reasoning and text-to-image generation. However, their reliance on sample-level rewards introduces a key limitation because uniform credit assignment across all tokens fails to capture fine-grained, token-level contributions. To address this issue, we propose Guidance-Contrastive Policy Optimization (GCPO), a novel algorithm that enables per-token credit assignment by contrasting model predictions under positive and negative prompts. Rather than uniformly broadcasting sample-level advantages, GCPO assigns token-level advantages proportional to the difference between these contrastive predictions, allowing more precise and informative learning signals. Empirically, we find that GCPO emphasizes semantically relevant regions—such as visual areas aligned with textual prompts in text-to-image generation, and critical keywords within reasoning traces for chain-of-thought tasks. Through extensive experiments, GCPO consistently outperforms GRPO and DAPO baselines on both text-to-image generation and chain-of-thought reasoning benchmarks, demonstrating its effectiveness as a general and scalable optimization strategy for discrete policy learning.

Primary area:
reinforcement learning
Keywords:
text-to-image, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SMKVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies token credit assignment in GRPO/DAPO and proposes GCPO. The method uses the KL divergence between predictive distributions under positive and negative conditioning to construct token weights, applies rank-based normalization, and uses the normalized weights to scale the per-sample advantage. Experiments on text-to-image generation, multimodal reasoning, and mathematical reasoning show some improvements over the evaluated baselines.

Critical strengths
  1. The proposed method is simple, does not require training an additional value model or token reward model, and shows some improvements in text-to-image generation and reasoning tasks.
Critical weaknesses
  1. The credit assignment claim lacks direct support. The difference between predictive distributions under positive and negative prompts reflects how sensitive predictions are to the prompt conditions, but does not directly measure each token's contribution to the final reward. Since the weights are non-negative, token advantages within the same trajectory cannot have opposite signs. Correct and incorrect steps may therefore be rewarded or penalized together, which does not establish more accurate credit assignment.
  2. The interpretation of some experimental results overstates the contribution of token weighting. The Counting score increases from 0.59 for the base model to 0.84 for GCPO, but the reproduced GRPO baseline already reaches 0.81. The gain over this baseline is 0.03, so the entire improvement cannot be attributed to the added token weighting mechanism.
Questions for authors
  1. In Table 8, the reported share of reasoning tokens increases from 1.9% to 18.1%. Are all columns computed using the same set of rollouts and the same aggregation method? If the final rank-based weights are directly aggregated and their mean is approximately 0.5, with a maximum of at most 1, how can this degree of amplification be explained? Please specify the exact quantity being measured and how it is computed.
  2. Is the probability ratio used for training computed from the original conditional distribution or the actual sampling distribution after CFG? If the two distributions differ, how is the mismatch between the behavior policy and the target policy handled? Do GRPO and GCPO use the same implementation?
1 comment

Q1

We use fixed rollout from the base checkpoint, we report tokens' weighting contribution, defined as

sums of weight of tokens of a particular classsums of weight of all tokens.\frac{\text{sums of weight of tokens of a particular class}}{\text{sums of weight of all tokens}}. Note this value is not “average weight”, but instead the “ratio” of total contribution. For example, given two token class A,B, and a 3-token sequence whose token classes are [A,B,B],one possible weight under rank normalization is [0.0,0.5,1.0]. Another is [1.0,0.5,0.0]. Comparing the two setup, the contribution ratio for class a jumps from 0% to 66.6%. Hence, increases from 1.9% to 18.1% from Table 8 is reasonable. We also note that the uniform baseline would be have a weight of [1,1,1], and class A's contribution is 33.3% under this scheme, which is just the token frequency.

Q2

We use original distribution during training and use CFG only for rollout because including CFG in RL training leads to instability. Qwen-Image-RL [1] reported similar issues

W1 Justification

We agree that correct and incorrect steps may be rewarded or penalized together. However, this limitation does not invalidate the underlying credit assignment. Existing token-level credit assignment methods, such as [2] and VPPO, operate under the same constraint. Moreover, vanilla GRPO has demonstrated that uniformly propagating sample-level rewards to all tokens can already yield performance improvements. Rather than distinguishing correct from incorrect tokens, token-level credit assignment methods investigate where the learning signal should be concentrated. For example, VPPO emphasizes visually dependent tokens, [2] focuses on high-entropy tokens, and GCPO identifies tokens that are sensitive to contrastive signals.

As clarified in our response to Q1, we report the relative contributions to the total learning weight under different weighting paradigms:

  • Uniform: a weight of 1 for all tokens; this effectively reports token frequencies in the rollouts
  • GCPO-KL and GCPO-IG: contrastive weighting with negative prompts, using the KL and IG metrics, respectively
  • VPPO: contrastive KL weighting with a noise-corrupted image input
  • Entropy: pure entropy-based weighting in [2]

The metrics are computed using base model's rollouts.

GroupuniformGCPO-KLGCPO-IGVPPOEntropy
math_symbolic13.8%21.8%21.5%9.4%2.0%
math_semantic2.3%13.1%13.6%2.9%16.4%
reasoning1.9%18.1%16.4%3.3%45.5%
content_words63.1%36.5%36.3%64.5%25.3%
filler18.9%10.6%12.2%19.9%10.8%

We bold all numbers that are at least 150% of the uniform baseline, indicating that they are heavily upweighted.

Overall, compared with the baseline frequency (uniform), GCPO amplifies the learning signal primarily for math_symbolic, math_semantic, and reasoning tokens.

Comparison to Entropy: Both methods emphasize reasoning connectives such as “however” and “finally.” These tokens have already been shown in [1] to be high-entropy “forking tokens” that significantly affect learning. However, GCPO’s amplification is smoother (1.9% → 18.1% vs. 1.9% → 45.5%), and it also emphasizes low-entropy symbolic math tokens, which we argue can be equally important.

Comparison to VPPO: GCPO follows a completely different distribution from VPPO. VPPO does not heavily emphasize reasoning, math_symbolic, or math_semantic tokens. Its weight is concentrated on “content_words.” Through qualitative examination, we find that these tokens are mostly OCR text describing the image.

Importantly, the improvements do not arise from explicitly identifying correct and incorrect tokens. Rather, they arise from concentrating learning signals on task and reasoning-relevant tokens. Updating model predictions at these positions can have a greater impact on task performance than updating probabilities associated with fillers, punctuation, and other less informative tokens.

Regarding the assignment of negative credit to incorrect reasoning steps, we agree that this is a promising direction. Such fine-grained feedback is typically provided by process reward models (PRMs), which evaluate individual reasoning steps using partial reasoning traces. However, this falls into a complementary line of research and is outside the scope of our work. Our focus is on the setting where only sample-level rewards are available, without additional step-level supervision.

W2 Entire improvement cannot be attributed to the added token weighting mechanism.

We have updated the relavent section (4.1) with a more accurate analysis on the per category improvements. The section now focuses on color attribution category, which benefits from region-aware GCPO significantly.

[1] Qwen-Image-2.0-RL Technical Report

[2] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning

Reviewer_TCtrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GCPO for token-level credit assignment in discrete policy optimization, which computes KL divergence between prediction distributions under positive and negative prompts, applies rank-based normalization, and uses the resulting token weights to scale sample-level advantages. The authors evaluate GCPO on text-to-image generation, multimodal reasoning, and math reasoning, showing improvements over GRPO and DAPO.

Critical strengths
  1. The paper uses the prediction differences under positive and negative conditioning in CFG for token weighting during training, encouraging optimization to focus on prompt-relevant image regions or reasoning tokens. This is a valuable idea.

  2. Histogram equalization is a practical design choice, which reduces the dependence of token weights on the absolute scale of KL divergence and mitigates weight concentration caused by outlier tokens.

  3. The experiments extend beyond text-to-image generation to reasoning tasks, for which the paper construct corresponding negative prompts.

Critical weaknesses
  1. While using negative prompts is motivated for image generation, uniformly appending “Generate the wrong answer” to construct negative prompts for language tasks is less convincingly justified, particularly when compared with VPPO’s design.

  2. Prior work [1] has investigated critical-token selection and dynamic advantage weighting for autoregressive image generation, but this paper neither discusses nor compares with it.

[1] Group Critical-token Policy Optimization for Autoregressive Image Generation

Questions for authors
  1. Table 5 demonstrates a performance advantage over several alternative negative conditions, but the rationale behind this construction requires a more convincing explanation and further discussion.

  2. Adding comparison with previous work [1] would help further establish the effectiveness of the proposed contrastive guidance.

  3. Have the paper compared the effects of using an empty (unconditional) prompt versus an explicit negative prompt for contrastive guidance in T2I task?

[1] Group Critical-token Policy Optimization for Autoregressive Image Generation

1 comment

W1 Q1 Justification

We refer the reviewer to appendix D for a detailed discussion. Given fixed rollout from the base model, we evaluate each token's weighting contribution under different schemes, which are defined as

sums of weight of tokens of a particular classsums of weight of all tokens.\frac{\text{sums of weight of tokens of a particular class}}{\text{sums of weight of all tokens}}. we categorize tokens into the following classes:

  • Math_symbolic: digits, mathematical symbols (e.g., + and -), LaTeX expressions (e.g., \sqrt{}), and single letters (e.g., x and y)
  • Math_semantic: 71 common mathematical keywords (e.g., solve, equation, integral, fraction, and one)
  • Reasoning: 23 common reasoning connectives (e.g., thus, therefore, however, finally, and recall)
  • Filler: 40 common fillers, linking verbs, and prepositions (e.g., do, does, is, are, in, on, and with)
  • Generic: all other text

We report their relative contributions to the total learning weight under different weighting paradigms:

  • Uniform: a weight of 1 for all tokens; this effectively reports token frequencies in the rollouts
  • GCPO-KL and GCPO-IG: contrastive weighting with negative prompts, using the KL and IG metrics, respectively
  • VPPO: contrastive KL weighting with a noise-corrupted image input
  • Entropy: pure entropy-based weighting in [2]

The metrics are computed using base model's rollouts.

GroupuniformGCPO-KLGCPO-IGVPPOEntropy
math_symbolic13.8%21.8%21.5%9.4%2.0%
math_semantic2.3%13.1%13.6%2.9%16.4%
reasoning1.9%18.1%16.4%3.3%45.5%
content_words63.1%36.5%36.3%64.5%25.3%
filler18.9%10.6%12.2%19.9%10.8%

We bold all numbers that are at least 150% of the uniform baseline, indicating that they are heavily upweighted.

Overall, compared with the baseline frequency (uniform), GCPO amplifies the learning signal primarily for math_symbolic, math_semantic, and reasoning tokens.

Comparison to Entropy: Both methods emphasize reasoning connectives such as “however” and “finally.” These tokens have already been shown in [1] to be high-entropy “forking tokens” that significantly affect learning. However, GCPO’s amplification is smoother (1.9% → 18.1% vs. 1.9% → 45.5%), and it also emphasizes low-entropy symbolic math tokens, which we argue can be equally important.

Comparison to VPPO: GCPO follows a completely different distribution from VPPO. VPPO does not heavily emphasize reasoning, math_symbolic, or math_semantic tokens. Its weight is concentrated on “content_words.” Through qualitative examination, we find that these tokens are mostly OCR text describing the image.

In conclusion, GCPO emphasizes symbolic math, semantic mathematical keywords, and logical connectives, which we believe are important. The difference between KL and information gain is small; the most important component is the contrastive construction.

W2 Q2 Additional Baselines and References

As requested, we add additional baselines in Table 1 in the revision and added citation and discussions in the related work session. We reproduce [2] under comparable evaluation settings (4 samples / prompt) using the official checkpoint. Results show that GCPO outperforms [2] (0.89 vs 0.88 GenEval). We further conducted uncertainty estimate by sweeping 3 evalation seeds, the results are summarized below

-[2]GCPO
GenEval0.88 ±\pm 0.000.89 ±\pm 0.00

We note that we were unable to reproduce the reference number 0.90 reported by [2], whose sampling setup like temperature and samples/prompt are unspecified. In our reproduction, the best seed has a score of 0.8829 while the worst seed has a score of 0.8825. This low variance is expected since the benchmark portocal already uses avg@4 score.

W3 Q3 Explicit negative prompt for contrastive guidance in T2I task

We conducted additional study by using the negative prompt "deformed, low quality, blurry". Results are shown below

-Negative PromptNull Prompt (GCPO)
GenEval0.85 ±\pm 0.000.89 ±\pm 0.00

Explict negative prompts is worse becasue the base model is not trained on such prompt distribution and cannot understand its meaning. We also evaluate the base model's performance with different sampling guidance

-GenEval
Janus-Pro-7B + CFG (Negative Prompt)0.77
Janus-Pro-7B + CFG (Null Prompt)0.80

This suggests negative prompt may lead to worse sampling distribtuion at the start of training, affecting the learning outcome. The biggest drop for the two base model evalautions is on the color attribution category (0.66 vs 0.60). However, we note that under negative prompt, GCPO still improves performance ( 0.77-> 0.85).

[1] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning [2] Group Critical-token Policy Optimization for Autoregressive Image Generation

Reviewer_CT2FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Guidance Contrastive Policy Optimization (GCPO), an approach to improve token-level credit assignment in GRPO. Instead of assigning the same advantage to all tokens in a generated sequence, the proposed approach computes token-level weights using the KL divergence between model predictions conditioned on positive and negative prompts. These weights are then normalized and used to scale the sequence-level advantages during policy optimization. The authors evaluate GCPO on text-to-image generation and reasoning tasks, and empirically show that GCPO outperforms GRPO and baselines on several benchmarks.

Critical strengths
  1. The paper has a clear motivation to improve token-level credit assignment in GRPO-based methods. The proposed approach is relatively simple and can be integrated into existing policy optimization methods without requiring an additional reward model.

  2. The papers provides good experimental evaluation, covering text-to-image generation and reasoning tasks. The authors also provide useful comparisons with relevant baselines and relevant ablations.

Critical weaknesses
  1. My main concern is the motivation behind the proposed token-level credit assignment. The method uses the KL divergence between predictions under positive and negative prompts to determine token importance weight. Although this measures how sensitive the model's predictions are to the input prompts, it is uncertain how much each token contributes to the final correctness reward and whether these tokens actually contribute more to successful reasoning. I think the authors should better justify this.

  2. Another concern is whether the performance improvements justify the additional computational cost. Although GCPO improves over DAPO, the gains over the stronger VPPO baseline are relatively small on several benchmarks (e.g. on Qwen3-VL-8B, MathVerse and MathVision). Since the proposed method requires an additional model evaluation using negative prompts, I think the authors should provide a clearer comparison of training costs and performance improvements.

Questions for authors
  1. Can the authors provide further justification or analysis showing that the contrastive KL scores actually reflect how much each token contributes to the final reward?

  2. Can the authors report the additional computational cost of GCPO compared to DAPO and VPPO? It would also be useful to see how these methods compare under the same computational budget.

  3. Are the results averaged over multiple training seeds? If not, can the authors report the mean, variance etc across seeds?

1 comment

W1 Q1 Justification

We refer the reviewer to appendix D for a detailed discussion. Given fixed rollout from the base model, we evaluate each token's weighting contribution under different schemes, which are defined as

sums of weight of tokens of a particular classsums of weight of all tokens.\frac{\text{sums of weight of tokens of a particular class}}{\text{sums of weight of all tokens}}. we categorize tokens into the following classes:

  • Math_symbolic: digits, mathematical symbols (e.g., + and -), LaTeX expressions (e.g., \sqrt{}), and single letters (e.g., x and y)
  • Math_semantic: 71 common mathematical keywords (e.g., solve, equation, integral, fraction, and one)
  • Reasoning: 23 common reasoning connectives (e.g., thus, therefore, however, finally, and recall)
  • Filler: 40 common fillers, linking verbs, and prepositions (e.g., do, does, is, are, in, on, and with)
  • Generic: all other text

We report their relative contributions to the total learning weight under different weighting paradigms:

  • Uniform: a weight of 1 for all tokens; this effectively reports token frequencies in the rollouts
  • GCPO-KL and GCPO-IG: contrastive weighting with negative prompts, using the KL and IG metrics, respectively
  • VPPO: contrastive KL weighting with a noise-corrupted image input
  • Entropy: pure entropy-based weighting in [2]

The metrics are computed using base model's rollouts.

GroupuniformGCPO-KLGCPO-IGVPPOEntropy
math_symbolic13.8%21.8%21.5%9.4%2.0%
math_semantic2.3%13.1%13.6%2.9%16.4%
reasoning1.9%18.1%16.4%3.3%45.5%
content_words63.1%36.5%36.3%64.5%25.3%
filler18.9%10.6%12.2%19.9%10.8%

We bold all numbers that are at least 150% of the uniform baseline, indicating that they are heavily upweighted.

Overall, compared with the baseline frequency (uniform), GCPO amplifies the learning signal primarily for math_symbolic, math_semantic, and reasoning tokens.

Comparison to Entropy: Both methods emphasize reasoning connectives such as “however” and “finally.” These tokens have already been shown in [1] to be high-entropy “forking tokens” that significantly affect learning. However, GCPO’s amplification is smoother (1.9% → 18.1% vs. 1.9% → 45.5%), and it also emphasizes low-entropy symbolic math tokens, which we argue can be equally important.

Comparison to VPPO: GCPO follows a completely different distribution from VPPO. VPPO does not heavily emphasize reasoning, math_symbolic, or math_semantic tokens. Its weight is concentrated on “content_words.” Through qualitative examination, we find that these tokens are mostly OCR text describing the image.

Rather than distinguishing correct from incorrect tokens, token-level credit assignment methods investigate where the learning signal should be concentrated. For example, VPPO emphasizes visually dependent tokens, [2] focuses on high-entropy tokens, and GCPO identifies tokens that are sensitive to contrastive signals. The improvements do not arise from explicitly identifying correct and incorrect tokens. Rather, they arise from concentrating learning signals on task and reasoning-relevant tokens. Updating model predictions at these positions can have a greater impact on task performance than updating probabilities associated with fillers, punctuation, and other less informative tokens.

GCPO emphasizes symbolic math, semantic mathematical keywords, and logical connectives, which we believe are important. The difference between KL and information gain is small; the most important component is the contrastive construction.

W2 Q2. Compute

Token weights have no attached gradients, so the scoring cost is one no-gradient forward pass. We observed no meaningful differences from DAPO. DAPO, VPPO, and GCPO all achieve 5 steps/hour with Qwen3-VL-8B and 24 with Qwen2.5-VL. Cost is dominated by rollouts and Ray worker context switching. Qwen3-VL-8B is slower due to longer rollouts and more frequent DAPO online filtering. The VPPO paper reports a 10% speed difference from DAPO, which we did not observe, possibly due to infrastructure differences.

Q3. Seed

Results in the original submission are not averaged over multiple seeds. In particular, we based our implementation on VPPO codebase and uses the same training setup. We directly cite the baselines. We conducted additional runs on text-to-image and report the three seed’s result below

SeedSingleTwoCountingColorPositionColor Attri.Overall
#11.000.950.840.890.830.830.89
#21.000.930.910.870.880.760.89
#31.000.950.830.880.830.840.89

[1] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning

Comments on the paper

1 comment

We thank the reviewers for constructive comments and have updated the submission to address these concerns. Unfortunately, due to the lack of engagement in the rebuttal discussion period, the authors have decided to withdraw from ICLR at this time. We will continue to improve manuscript and submit to a future venue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.