acceptodds
Under review as a conference paper at ICLR 2027

The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Credits

Abstract

A commonly accepted explanation of critic-free RL (e.g., GRPO) for LLMs, based on sequence-level rewards, is that it reinforces successful rollouts with a positive advantage while penalizing failed ones. In contrast, we study critic-free RL from a token-level perspective, revealing the token flipping phenomenon: positive and negative rollouts exhibit remarkably similar proportions of tokens whose probabilities are boosted and suppressed during RL training. To explain this phenomenon, we further show that a token's change in probability is not fully determined by its own advantage; coupled gradient interactions with other tokens also play a non-negligible role. Specifically, these token coupling effects occur primarily between identical tokens that are both predicted with low confidence. Building upon this analysis, we propose the cancellation hypothesis: as a result of coupling, opposing signals cancel out for tokens shared by positive and negative rollouts, while tokens more specific to successful rollouts receive stronger reinforcement, thereby inducing hidden token-level credit assignment from rollout-level rewards. We support this hypothesis with complementary empirical evidence. (1) Compared with training on only positive rollouts, critic-free RL shifts updates from template and formatting tokens toward reasoning tokens; (2) Tokens boosted by critic-free RL consistently demonstrate higher value than suppressed tokens, regardless of whether they originate from positive or negative rollouts. Guided by this view, we implement two batching interventions to encourage or preserve cancellation in critic-free RL training: query-preserved mini-batching, which keeps rollouts from the same prompt together to encourage token coupling, and reward-balanced batching, which ensures that positive and negative rollouts are balanced in each update. Despite their simplicity, these interventions improve RLVR training across multiple model scales, supporting cancellation as both an explanatory principle and a practical design criterion for critic-free RL training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_t3ePOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies critic-free reinforcement learning (such as GRPO) from the perspective of token credits. The authors first discover and investigate the token flipping phenomenon in GRPO, where positive and negative rollouts exhibit similar boosted or suppressed token ratios. They then attribute this to token coupling, i.e., gradient interactions between tokens. The authors subsequently propose the cancellation hypothesis, which states that the coupling signals of identical tokens in both positive and negative rollouts cancel each other out within a batch. The authors empirically verify this hypothesis and propose two training strategies, query-preserved mini-batching and reward-balanced batching. Extensive experiments validate the effectiveness of the proposed methods.

Critical strengths
  1. This paper has a clear and logical structure, and a complete and meticulous narrative logic, making it easy to follow.
  2. Token flipping is a very novel and interesting discovery that offers a fresh perspective on the token credits issue in critic-free RLs such as GRPO. Figure 1, with its ingenious design, clearly illustrates this phenomenon, demonstrating considerable insight. The authors subsequently validate their analysis with theoretical and empirical evidence.
  3. The two training strategies proposed by the authors have brought consistent and significant improvements on different models and benchmarks, including some tasks that are difficult for relatively small models.
Critical weaknesses
  1. The token flipping phenomenon the authors discovered is interesting, but its explanation is not new. For example, the derivation in Section 3.1 is a standard characteristic of all policy gradient methods with baseline subtraction and is not unique. The paper does not show what is specific to GRPO.
  2. There are some potentially overly strong assumptions and settings that do not conform to reality. For example, the authors claim in Equation 12 that "in the idealized case where the common within-query overlap is approximately a positive constant," which may be inconsistent with reality and lacks necessary empirical evidence. More importantly, the authors use the output-layer as an analysis proxy, and its sign consistency with the full model direction is only about 70%, which leaves roughly 30% disagreement ("random chance" is 50%). In addition, the optimizer (SGD) used in the analysis is inconsistent with the one used for real-world training, although the authors provide some explanation in the appendix. One-step linearization is used to support claims about long-run dynamics, and the empirical setting in the appendix, which repeatedly updates on the same batch of data, is also very strange.
  3. The proposed method may be very costly. For example, for reward-balanced batching, the cost could be extremely high (e.g., with very low or very high accuracy). Furthermore, discarded samples may contain valuable information that is being wasted. Additionally, dynamic sampling is a very standard practice, somewhat similar to the authors' method, which they do not compare.
Questions for authors

See Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to reviewer t3eP part 1/n

W1. Scope beyond GRPO

The token flipping phenomenon the authors discovered is interesting, but its explanation is not new. For example, the derivation in Section 3.1 is a standard characteristic of all policy gradient methods with baseline subtraction and is not unique. The paper does not show what is specific to GRPO.

Thank you for raising this point. We agree that the covariance identity in the original Sec. 3.1 follows from centering the advantages, and we have moved the derivation to Appendix E.3. Furthermore, we do think that Cancellation can arise beyond GRPO when rollout gradients with opposite weights share directions. We observe a similar pattern with REINFORCE and a global baseline: template tokens account for 32.4% of the log-probability boost with REINFORCE, compared with 51.2% and 52.2% in the positive-only and negative-only training, respectively (Table 6 in our revised manuscript).

Our contribution could be summarized as characterization of how token coupling shapes implicit credit assignment, including (1) A token’s log-probability change depends on the contributions of other tokens in the batch, so the direction of its update can oppose its own rollout’s advantage. (2) Our analysis predicts strong coupling between identical, low-confidence tokens, which we validate empirically (Table 1 and Fig. 3(b)). (3) This motivates our cancellation hypothesis: opposing signals can cancel along directions shared by successful and unsuccessful rollouts, leaving more selective updates. Consistent with this interpretation, boosted tokens have higher Monte Carlo estimated value than suppressed tokens for both rollout advantage signs (Fig. 5(a)).

We chose GRPO as our baseline because it is widely used for critic-free RL method.

W2. Assumptions and analytical settings

W2.1. Within-query gradient overlap

There are some potentially overly strong assumptions and settings that do not conform to reality. For example, the authors claim in Equation 12 that "in the idealized case where the common within-query overlap is approximately a positive constant," which may be inconsistent with reality and lacks necessary empirical evidence.

Thank you for raising this concern. Our intuition is that correct and incorrect rollouts for the same query can draw on similar knowledge and share intermediate reasoning steps, which motivates our expectation of positive overlap between their response-level gradients.

To examine this intuition empirically, we measured rollout-gradient cosine similarity for 20 queries with 8 rollouts each (160 rollouts in total). We find that 94.82% of within-query pairs have positive gradient cosine similarity. The mean gradient cosine similarity is 0.191 within queries, compared with 0.131 across queries.

These measurements show that within-query gradient overlap is predominantly positive. We use a common positive constant cqc_q as an idealized approximation to illustrate the cancellation mechanism. The derivation, originally in Sec. 3.1 of the submission, now appears in Appendix E.3 of the revised manuscript, where we report these measurements in a footnote.

W2.2. Validity of the output-layer proxy

More importantly, the authors use the output-layer as an analysis proxy, and its sign consistency with the full model direction is only about 70%, which leaves roughly 30% disagreement ("random chance" is 50%).

Thank you for raising this concern. We use the output-layer proxy in Sec. 2 mainly to identify where coupling is strongest. The proxy predicts that coupling concentrates on identical, low-confidence token pairs (Eq. 10).

More importantly, the pattern predicted under such assumption are further validated with full parameter updates. Table 1 shows that masking tokens that meet both conditions yields similar Boost Rates (74.12% and 75.00%) and Mean Boosts (+0.0301 and +0.0306) under both LM-head-only and full-parameter updates, respectively. Figure 3(b) also shows that Boost Rate increases with predicted coupling strength in both settings. The new results with Adam in Table 1 show the same pattern. Masking identical, low-confidence tokens yields the highest Boost Rate and Mean Boost in both settings. In addition, all other analyses and interventions in Secs. 3–4 use the full model.

Note that related work also uses restricted gradient proxies for token-level analysis. OTB uses logit gradients to approximate token gradient norms [1], and DelTA uses LM-head gradients to estimate token coefficients [2].

[1] Li et al. The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL. ICML 2026. Paper.

[2] Zhang et al. DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards. 2026. arXiv:2605.21467.

AuthorsOpenReview ↗

Response to reviewer t3eP part 2/n

W2.3. SGD versus AdamW

In addition, the optimizer (SGD) used in the analysis is inconsistent with the one used for real-world training, although the authors provide some explanation in the appendix.

To answer reviewer's question, we clarify the scope of the SGD simplification and distinguish it from our empirical analyses and training experiments with AdamW.

SGD is mainly used for the derivation and the original causal verification. Specifically, we use SGD for the coupling derivation in Sec. 2 and the original experiments in Table 1 and Fig. 3. The token-flipping observations (Fig. 1), the empirical cancellation analyses (Figs. 4–7), and the QB/RB training experiments in Sec. 4 (Fig. 8 and Table 2) all are based on AdamW. In particular, Sec. 3 tests the cancellation hypothesis empirically under AdamW and motivates the batching interventions evaluated in Sec. 4, connecting the simplified analysis to practical training. Prior work has similarly considered SGD to analyze LLM training dynamics [1–4] and demonstrated its practical use in LLM-RL [5, 6].

To further bridge the gap, we have repeated the Table 1 experiment with Adam on Qwen3-1.7B-Base, using a batch from the 20th training step and retaining the optimizer's accumulated momentum. We evaluate 500 target tokens, masking 5 candidate tokens per target, under both LM-head-only and full-parameter updates:

Same tokenLow confidenceLM-head: Boost Rate (%)LM-head: Mean BoostFull: Boost Rate (%)Full: Mean Boost
——46.00+0.000053.30+0.0023
—✓80.30+0.003575.90+0.0402
✓—56.20-0.000350.40+0.0100
✓✓95.60+0.011586.90+0.1414

The same-token, low-confidence selection yields the highest Boost Rate and Mean Boost under both settings. These results provide direct evidence that the coupling structure identified in Sec. 2 also persists under Adam updates.

We also want to clarify why gradient cancellation could generally make sense under AdamW. Because contributions from tokens with positive and negative advantages are summed to form the batch gradient. Opposing components can therefore cancel before AdamW update its first- and second-moment estimates.

[1] Learning Dynamics of LLM Finetuning, ICLR 2025

[2] Deng et al. Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning. 2025. arXiv:2510.03669.

[3] On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization, NeurIPS 2025

[4] On GRPO collapse in search-r1: The lazy likelihood-displacement death spiral, ICML 2026

[5] Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs, ICML 2026

[6] Score Centering Stabilizes Off-policy Reinforcement Learning

W2.4. One-step analysis and long-run dynamics

One-step linearization is used to support claims about long-run dynamics, …

The derivation in Sec. 2 indeed describe a single update and motivates the cancellation hypothesis. We then test its implications for token-level credit assignment empirically under longer training. Specifically, our Monte Carlo token-value analysis (Sec. 3, Fig. 5(a)) shows that boosted tokens have higher estimated value than suppressed ibes, regardless of their rollout's reward sign. This training observation provides empirical support for the token-level credit-assignment interpretation of our cancellation hypothesis.

W2.5. Repeated updates on the same batch

… and the empirical setting in the appendix, which repeatedly updates on the same batch of data, is also very strange.

Thank you for raising this question. In the experiment in Appendix E.2 (Fig. 12), we deliberately keep the batch fixed in order to examine whether the value gap we observed in Fig. 5(a) becomes more pronounced with repeated updates. Fig. 12 shows that the mean value gap between boosted and suppressed tokens increases overall with repeated updates, providing further empirical support for our cancellation hypothesis. We have clarified this motivation in Appendix E.2.

AuthorsOpenReview ↗

Response to reviewer t3eP part 3/n

W3. RB sampling cost and comparison with dynamic sampling

The proposed method may be very costly. For example, for reward-balanced batching, the cost could be extremely high (e.g., with very low or very high accuracy). Furthermore, discarded samples may contain valuable information that is being wasted. Additionally, dynamic sampling is a very standard practice, somewhat similar to the authors' method, which they do not compare.

Thank you for raising this concern. We agree that RB can introduce additional sampling overhead per training step, especially when rewards are very sparse or saturated. However, we want to highlight that this overhead is adjustable through the threshold τ\tau. Lowering τ\tau relaxes the balancing requirement and can reduce the number of additional rollouts needed, allowing practitioners to adjust accroding to their sampling budget.

Comparison with dynamic sampling. Dynamic sampling proposed to do oversampling on certain queries. We compare RB with DAPO on OLMo-3-1025-7B under matched rollout budgets. Overall is the average score across AIME24, AIME25, AMC, OlympiadBench, and Minerva.

Rollout budgetDAPO overall (%)RB overall (%)
100,00034.7237.43
200,00038.4141.78
250,00039.3342.58
280,00040.7643.42

RB achieves a higher overall score at all four matched rollout budgets. We observe the same advantage on Qwen2.5-Math-7B with τ=0.4\tau=0.4:

Rollout budgetDAPO overall (%)RB overall (%)
100,00035.1436.93
200,00037.1638.50
300,00039.0042.35
400,00039.0043.71

RB achieves a higher overall score at all four budgets. The corresponding training curve is included in Sec. 4 (Fig. 9(a)) of the main text. We also compare DAPO, QB, RB, and QB+RB on Qwen2.5-Math-7B across multiple random seeds:

MethodAIME24AIME25AMCOlympiadBenchMinervaMATH-500Overall
DAPO24.10 ± 1.4212.29 ± 0.7558.35 ± 2.0838.17 ± 1.3430.64 ± 1.2975.40 ± 1.3939.82 ± 0.91
QB24.26 ± 3.4111.52 ± 0.6560.93 ± 0.5640.27 ± 2.1033.93 ± 2.4678.60 ± 2.6041.59 ± 0.72
RB25.55 ± 1.8812.11 ± 1.0361.54 ± 1.3738.78 ± 1.0535.28 ± 1.6777.13 ± 0.6141.73 ± 0.58
QB + RB28.12 ± 0.7315.23 ± 0.8562.62 ± 1.3442.49 ± 2.2934.55 ± 1.3181.40 ± 1.3944.07 ± 0.83

Both QB and RB achieve higher mean scores than DAPO on five of the six benchmarks, while QB+RB improves all six.

Sample utilization. During the first 300 training steps, RB uses 307,200 of 445,440 generated rollouts for policy updates, giving a cumulative utilization of 68.97%. Utilization over steps 1–100, 101–200, and 201–300 is 50.25%, 76.92%, and 94.34%, respectively. After warm-up ends at around step 75, update deferrals become less frequent and sample utilization improves overall. Sec. 4 (Fig. 9(b)) compares the utilization curves of RB and DAPO dynamic sampling. As a possible extension, asynchronous training could retain positive and negative rollouts in separate buffers for balancing across batches, further reducing sample waste.

We have added the Qwen2.5-Math-7B sampling-budget and sample-utilization curves to Sec. 4 of the main text (Fig. 9). The OLMo budget comparison (Fig. 13) and multi-seed results (Table 7) are provided in Appendix F.

Finally, we would like to highlight that QB and RB are simple follow-up interventions motivated by our proposed cancellation hypothesis. They are intended primarily to test whether preserving or strengthening cancellation improves training, and their empirical benefits provide further support for this hypothesis.

[1] Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Paper.

Reviewer_iB7TOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes GRPO-style critic-free RLVR at the token level and makes three claims.

1. Token flipping (Fig. 1, Fig. 9). After a single GRPO step, positive- and negative-advantage rollouts contain similar fractions of boosted and suppressed tokens, with similar magnitudes. This largely disappears under positive-only training.

2. Coupling kernel explanation.

  • A first-order expansion (Eqs. 4–5) splits each token's Δlog p into a self term and a cross-token coupling term.
  • An output-layer proxy (Eqs. 6–10) factorizes the coupling into representation similarity × output-distribution similarity. This predicts that coupling concentrates on same-token, low-confidence pairs.
  • Masked-update experiments support the prediction (Table 1): Boost Rate ≈ 74–75% when masking such tokens, vs. ≈ 50% for random tokens.

3. Cancellation hypothesis (Sec. 3). Directions shared by the positive and negative rollouts of a query cancel, while success-specific directions survive (Eqs. 11–13). Supporting evidence:

  • GRPO shifts boosted mass from template tokens toward reasoning tokens relative to positive-only training (Fig. 4).
  • Boosted tokens have higher Monte Carlo counterfactual value than suppressed tokens (Fig. 5a).
  • The value gap grows with sampling budget (Fig. 6).

Interventions (Sec. 4). The paper proposes query-preserved mini-batching (QB) and reward-balanced batching (RB). On DAPO-17K with three Qwen backbones, QB+RB raises the six-benchmark average from 37.8→45.0 (Qwen2.5-Math-7B), 31.9→36.1 (Qwen3-1.7B-Base), and 49.1→50.8 (Qwen3-4B-Base) (Table 2).

Critical strengths
  1. A well-designed, causal-style test of the coupling structure. The masked-update experiment (Table 1, Fig. 3) does more than show correlation:
    • Removing tokens predicted to be coupled (same token and low confidence) shifts the candidate update in a consistent direction.
    • The effect scales with predicted coupling strength (Fig. 3b,c).
    • LM-head-only and full-parameter updates agree closely (74.12/+.0301 vs. 75.00/+.0306).
  2. Simple, mechanism-motivated interventions with large effects in an unstable regime. QB fixes a concrete implementation issue: splitting a group across mini-batches breaks ∑iA^i=0\sum_i \hat{A}_i = 0 (Sec. 4, "Local imbalance"). RB keeps the reward-sign composition balanced. Together they stabilize the collapsing Qwen2.5-Math-7B run (Fig. 8) and raise its average by 7.2 points.
Critical weaknesses

1. The causal story is established under SGD, but the reported gains come from Adam training, and the paper does not connect the two. Eqs. 3–5 derive the coupling kernel from a plain gradient step θ′=θ+η∇θJ\theta' = \theta + \eta \nabla_\theta J, giving Kjk=⟨gj,gk⟩K_{jk} = \langle g_j, g_k \rangle. Under Adam the relevant kernel is instead KjkAdam=gj⊤PtgkK^{\text{Adam}}_{jk} = g_j^\top P_t g_k, with a time-varying preconditioner PtP_t, and the paper itself notes that Adam's early updates are close to sign descent and "wash out" per-token magnitudes (Appendix D.3). This is why the masked-update experiments switch to SGD with η=0.1\eta = 0.1 (Appendix D.2). As a result, the mechanism is validated in a regime different from the one in Table 2. Separately, the output-layer proxy agrees in sign with the full-parameter update on only about 70% of tokens (Fig. 10; App. D.1), which is a weak basis for calling it "faithful." An analysis or ablation under Adam, even approximate, would make the claim that cancellation explains the QB/RB gains much more convincing.

2. "Cancellation" is not separated from the standard group-baseline effect, and the empirical gains lack basic controls. Eq. 13 (surviving signal == covariance of the advantage with the loading on a direction) holds for any baseline-subtracted policy gradient, independent of the cross-token coupling in Sec. 2. The positive-only control A⋅1[r=1]A \cdot \mathbb{1}[r = 1] changes several things at once: it drops all negative-advantage rollouts (so the group advantages no longer sum to zero), reduces the effective data by the failure fraction, and rescales the update norm. Empirically, Table 2 reports no seed count or variance (apparently a single run per cell), the Qwen3-4B-Base gain is only 1.7 points with some per-benchmark drops (e.g., AIME25 24.2→22.824.2 \to 22.8 with QB; Minerva 44.5→43.444.5 \to 43.4 with QB+RB), the 7B baseline collapses (Fig. 8), DAPO dynamic sampling, the natural alternative to RB, is disabled (Appendix B), and the sample and compute cost of RB, which the authors acknowledge (Appendix A), is not quantified.

Questions for authors
  1. Does the Table 1 same-token/low-confidence effect hold when the masked update uses the actual Adam state from mid-training (not the first step, and not SGD)?
  2. Can you separate the group-baseline effect from cross-token coupling? For example, compare a REINFORCE control with a global baseline, a norm-matched positive-only control, and the negative-only control mentioned in Sec. 3.2 (no results are shown for it).
  3. Can you report Table 2 over ≥3\geq 3 seeds, include a DAPO dynamic-sampling baseline, and plot curves matched by samples or compute for RB, including the fraction of rollouts discarded or deferred?
  4. Token flipping uses the threshold ϵ=10−6\epsilon = 10^{-6} on log-probabilities from two FSDP forward passes (Appendix C), and the numerical precision is not stated. What is the noise floor, e.g. two forward passes with no update? Do the flipping fractions persist with ϵ\epsilon set above it?
6 comments
AuthorsOpenReview ↗

Resoibse ti reviewer iB7T part 1/n

1. Connecting the SGD analysis to Adam training (W1 / Q1)

The causal story is established under SGD, but the reported gains come from Adam training, and the paper does not connect the two.

Does the Table 1 same-token/low-confidence effect hold when the masked update uses the actual Adam state from mid-training (not the first step, and not SGD)?

Thank you for raising this point. We would like to clarify that our empirical analysis in Sec. 3 could already be regarded as this connection. Specifically, we use SGD for the derivation in Sec. 2 and the original experiments in Table 1 and Fig. 3. In Sec. 3, we explicitly test the cancellation hypothesis under AdamW using Monte Carlo token-value estimates. Boosted tokens have higher estimated value than suppressed tokens for both rollout advantage signs (Fig. 5(a)). These AdamW observations support the token-level credit-assignment interpretation of cancellation and further motivate the QB/RB interventions evaluated in Sec. 4, providing an empirical connection between our simplified derivation and the actual training.

Following your suggestion, we have repeated the Table 1 experiment with Adam on Qwen3-1.7B-Base, using a batch from the 20th training step and retaining the optimizer's accumulated momentum. We evaluate 500 target tokens, masking 5 candidate tokens per target, under both LM-head-only and full-parameter updates:

Same tokenLow confidenceLM-head: Boost Rate (%)LM-head: Mean BoostFull: Boost Rate (%)Full: Mean Boost
——46.00+0.000053.30+0.0023
—✓80.30+0.003575.90+0.0402
✓—56.20-0.000350.40+0.0100
✓✓95.60+0.011586.90+0.1414

The same-token, low-confidence selection yields the highest Boost Rate and Mean Boost under both settings, providing direct evidence that the coupling structure identified in Sec. 2 persists under Adam. We have clarified the optimizer choices and added these results to Table 1 in Sec. 2 of the revised manuscript, with experimental details in Appendix D.2.

2. Scope and reliability of the output-layer proxy (W1)

Separately, the output-layer proxy agrees in sign with the full-parameter update on only about 70% of tokens (Fig. 10; App. D.1), which is a weak basis for calling it "faithful."

We use the output-layer proxy in Sec. 2 (Eq. 6) mainly to identify where coupling primarily occurs. The proxy predicts that appreciable coupling occurs mainly between identical, low-confidence tokens (Eq. 10).

More importantly, we test this prediction by comparing LM-head-only and full-parameter updates in the masking experiment in Table 1. The original SGD results show highly consistent effects under both settings (74.12% vs. 75.00% Boost Rate, and +0.0301 vs. +0.0306 Mean Boost). Fig. 3(b) further shows consistent trends in Boost Rate across different coupling strengths (token similarity) under both settings. The new Adam results reported above also show a consistent pattern.

Overall, the output-layer proxy serves as an analytical approximation used only for the coupling analysis in Eqs. (6)–(10). The resulting structural prediction is then tested through full-parameter experiments (Table 1, Fig. 3(b)). Our subsequent cancellation analysis and experiments (Sec. 3), as well as the proposed interventions (Sec. 4), use the full model and do not rely on this output-layer approximation. We have revised Sec. 2 and Appendix D.1 to clarify this scope and describe the proxy in terms of its structural predictions. Similar gradient proxies are also used for token-level analysis in previous works like OTB [1] and DelTA [2].

[1] Li et al. The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL. ICML 2026. Paper.

[2] Zhang et al. DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards. 2026. arXiv:2605.21467.

AuthorsOpenReview ↗

Reponse to reviewer iB7T part 2/n

3. Group-baseline effects and control experiments (W2 / Q2)

"Cancellation" is not separated from the standard group-baseline effect...

We agree that Eq. 13 in the submission does not require the particular coupling structure identified in Sec. 2. We have moved it to Appendix E.3 of the revised manuscript as supplementary interpretation.

Our main findings concern the token-level coupling structure and implicit credit assignment. Sec. 2 identifies a coupling structure between identical, low-confidence tokens, which we validate in Table 1 and Fig. 3. The empirical analyses in Sec. 3 show that boosted tokens have higher estimated value than suppressed tokens, regardless of their rollout's reward sign (Fig. 5(a)). These findings support the token-level credit-assignment interpretation of cancellation and remain in the main text.

Can you separate the group-baseline effect from cross-token coupling? For example, compare a REINFORCE control with a global baseline, a norm-matched positive-only control, and the negative-only control mentioned in Sec. 3.2 (no results are shown for it).

Following your suggestion, we complement the original GRPO results with REINFORCE with global baseline, and positive-only and negative-only controls. Following Fig. 4, the four token-category columns report each category's share of total log-probability boost (%):

Training SettingMathTemplateSemanticConjunctionGradient norm
Positive-only7.651.223.117.80.042488
Negative-only8.152.221.318.20.047448
REINFORCE with global baseline9.232.436.721.40.064373

Positive-only and negative-only updates allocate 51.2% and 52.2% of the boost to template tokens, respectively. Under REINFORCE with global baseline, the template share is 32.4%. This shift in learning focus follows the pattern observed with GRPO in Fig. 4 and is consistent with our cancellation interpretation. We have added these additional results to Appendix E.1 (Table 6).

4. Robustness across seeds and benchmarks (W2 / Q3)

Empirically, Table 2 reports no seed count or variance (apparently a single run per cell)…

Can you report Table 2 over ≥3\geq 3 seeds, include a DAPO…

Thank you for this suggestion. We have added experiments across training seeds and model families to further assess the robustness of the improvements.

More independent training runs. Repeating every setting in Table 2 would require more time and compute than available during the rebuttal. We therefore focus on Qwen2.5-Math-7B, the largest model in the original Table 2.

MethodAIME24AIME25AMCOlympiadBenchMinervaMATH-500Overall
Baseline22.48 ± 1.8211.64 ± 1.0558.27 ± 3.2337.09 ± 1.0828.43 ± 1.1271.87 ± 1.1038.32 ± 0.88
DAPO24.10 ± 1.4212.29 ± 0.7558.35 ± 2.0838.17 ± 1.3430.64 ± 1.2975.40 ± 1.3939.82 ± 0.91
QB24.26 ± 3.4111.52 ± 0.6560.93 ± 0.5640.27 ± 2.1033.93 ± 2.4678.60 ± 2.6041.59 ± 0.72
RB25.55 ± 1.8812.11 ± 1.0361.54 ± 1.3738.78 ± 1.0535.28 ± 1.6777.13 ± 0.6141.73 ± 0.58
QB + RB28.12 ± 0.7315.23 ± 0.8562.62 ± 1.3442.49 ± 2.2934.55 ± 1.3181.40 ± 1.3944.07 ± 0.83

Relative to the baseline, QB and RB both also achieve higher mean scores than DAPO and baseline on five of the six benchmarks, while QB+RB improves all six relative to both the baseline and DAPO.

Additional model families. We also evaluate Llama-3.2-3B-Instruct and OLMo-3-1025-7B. The full results are included in Table 2 in the main text of the revised manuscript:

ModelBaselineQBRBQB + RB
Llama-3.2-3B-Instruct21.1123.0424.0124.17
OLMo-3-1025-7B49.6250.3551.7652.26

To highlight, we designed QB and RB to test whether preserving conditions for cancellation improves RL training. The gains in the multi-seed and cross-model experiments durther support this prediction. We have added the independent-training results to Appendix F (Table 7) and the additional model-family results to Table 2 in the main text. The matched-budget comparison between RB and DAPO on OLMo is presented in the next response and Appendix F (Fig. 13).

AuthorsOpenReview ↗

Reponse to reviewer iB7T part 3/n

5. DAPO comparison and RB sampling costs (W2 / Q3)

… DAPO dynamic sampling, the natural alternative to RB, is disabled (Appendix B), and the sample and compute cost of RB, which the authors acknowledge (Appendix A), is not quantified.

… include a DAPO dynamic-sampling baseline, and plot curves matched by samples or compute for RB, including the fraction of rollouts discarded or deferred?

To clarify, DAPO uses oversampling (e.g., 80 rollouts for certain queries and 8 rollouts for the other) to fill the training batch with query groups that have nonzero advantages [1]. RB applies the reward-balance threshold to the buffered batch. When the batch falls short of the threshold, we defer the policy update and continue collecting rollouts for other queries.

RB outperforms dynamic sampling under matched sampling budget. We compare RB with DAPO on OLMo-3-1025-7B under matched rollout budgets. The table below reports the mean score across AIME24, AIME25, AMC, OlympiadBench, and Minerva:

Rollout budgetDAPO overall (%)RB overall (%)
100,00034.7237.43
200,00038.4141.78
250,00039.3342.58
280,00040.7643.42

We observe the same pattern on Qwen2.5-Math-7B with τ=0.4\tau=0.4:

Rollout budgetDAPO overall (%)RB overall (%)
100,00035.1436.93
200,00037.1638.50
300,00039.0042.35
400,00039.0043.71

RB achieves a higher overall score at all four budgets on both models. The multi-seed results in our previous response part 2/n also show higher overall scores for QB, RB, and QB+RB than for DAPO.

Sample utilization along the training. We measured sample utilization on Qwen2.5-Math-7B with the RB threshold set to τ=0.4\tau=0.4. At each cumulative step count, all settings use the same number of rollouts for updates. The method columns show the number of finished rollouts and the fraction used for updates in parentheses.

First N stepsRollouts used for updates per settingDAPORB
100102,400254,528 (40.23%)203,776 (50.25%)
200204,800532,272 (38.48%)336,896 (60.79%)
300307,200786,808 (39.04%)445,440 (68.97%)

To show how RB utilization changes over training, we also report it separately for each 100-step interval.

Training stepsFinished rolloutsRollouts used for updatesRB utilization
1–100203,776102,40050.25%
101–200133,120102,40076.92%
201–300108,544102,40094.34%

After around step 75, training rewards improve, update deferrals become less frequent and sample utilization improves overall. As a possible extension, asynchronous training could retain positive and negative rollouts in separate buffers for balancing across batches, further reducing sample waste.

The whole Qwen2.5-Math-7B sampling-budget and sample-utilization curves are now included in Sec. 4 of the main text (Fig. 9). The OLMo matched-budget comparison (Fig. 13) and the multi-seed comparison on Qwen2.5-Math-7B (Table 7) are provided in Appendix F. We have also clarified the update condition in Appendix B.

[1] Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Paper.

Reviewer_4bKSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how critic-free RL produces selective token-level updates despite assigning a shared advantage to all tokens within a rollout. The authors identify “token flipping,” where token probabilities change against their rollout-level advantage signs, and explain this through a first-order decomposition into self-influence and cross-token gradient interactions. Using an output-layer approximation, they argue that coupling concentrates on identical, low-confidence tokens with aligned representations. They propose the cancellation hypothesis: opposing signals suppress directions shared by successful and unsuccessful rollouts, leaving more success-associated signals. This interpretation motivates query-preserved mini-batching and reward-balanced batching, which improve mathematical reasoning performance across three Qwen backbones.

Critical strengths
  1. The paper combines controlled token-masking experiments, comparisons with positive-only updates, and Monte Carlo estimates of token value. These analyses provide evidence beyond aggregate benchmark improvements and help connect gradient interactions to the usefulness of the resulting updates.

  2. The proposed batching strategies are simple and easy to implement

Critical weaknesses
  1. Some contributions overlap with previous works. While this work cited previous works like THR [1], contributions like low likelihood and large embedding similarity as influential tokens are not credited, whereas THR [1] also casually shows that training with those tokens drives most of the learning. Thus, these contributions should be clearly discussed.

  2. The experiment is only on the Qwen model; more model families like Llama or Numotron should be included.

  3. Preserving complete query groups may restrict token-budget balancing when rollout lengths vary substantially, potentially increasing workload imbalance or padding overhead depending on the implementation. Reward-balanced buffering can additionally reduce sample utilization and increase update latency.

I'm willing to raise my score if the concerns are resolved.

[1] Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning

Questions for authors
  1. The contrast is limited in on-policy learning; however, when efficiency is required in RL, often the rollouts can be off-policy in practice due to efficiency requirements, which can lead to low, how the off-policy effect could be studied to further extend the view of the paper.
5 comments
AuthorsOpenReview ↗

Response to reviewer 4bKS part 1/n

W1. Relationship to THR

Some contributions overlap with previous works. While this work cited previous works like THR [1], contributions like low likelihood and large embedding similarity as influential tokens are not credited, whereas THR [1] also casually shows that training with those tokens drives most of the learning. Thus, these contributions should be clearly discussed.

Thank you for highlighting this connection: our output-layer coupling kernel shares the mathematical structure of the interaction term in THR [1]. We agree that this shared analytical basis should be acknowledged more explicitly.

For two tokens uu and vv, the coupling kernel we share with THR takes the following form.

Kv,u(W)=⟨hv,hu⟩⟨eov−πv,eou−πu⟩.K^{(W)}_{v,u} = \langle h_v, h_u \rangle \langle e_{o_v} - \pi_v, e_{o_u} - \pi_u \rangle.

Here hh is the hidden representation, π\pi is the next-token distribution, and eoe_o is the one-hot vector of the sampled token. The table below summarizes the differences in research questions, quantities of interest, underlying intuitions, and observed sign patterns.

AspectTHROur work
Research question“how to explicitly steer training toward exploration or exploitation” ([1], Abstract, p. 1).How do shared rollout-level advantages produce selective updates to individual tokens?
Quantity of InterestFix a training token uu and aggregate its coupling effect with positive reference rollouts: THRu=su∑i∈R+(qu)1Ti∑v∈yiKv,u(W)\mathrm{THR}_u=s_u\sum_{i\in\mathcal R^+(q_u)}\frac{1}{T_i}\sum_{v\in y_i}K^{(W)}_{v,u} ([1], Definition 4.1 and Corollary 4.2, p. 4; rewritten in our notation).Fix an observed token vv and aggregate the coupling effects from all training tokens in the batch: Δlog⁡pv≈ηN∑u∈BAuKv,u(W)\Delta\log p_v\approx\frac{\eta}{N}\sum_{u\in\mathcal B} A_uK^{(W)}_{v,u}.
Underlying intuitionUse the change in correct-response log-likelihood “as a proxy for the GRPO objective”; summing a training token's interactions with positive reference tokens quantifies its contribution to that change ([1], Sec. 4 opening paragraph, Definition 4.1 and Corollary 4.2, p. 4).Opposing training signals can cancel shared gradient components, leaving stronger updates along directions associated with reward differences.
Observed sign patternsBoth correct and incorrect rollouts contain tokens with positive and negative THR scores. The sign describes a training token's estimated contribution to the change in correct-response log-likelihood ([1], Fig. 2 and Sec. 5, pp. 4–5).We measure the change in each observed token's own log-probability after an update. Its sign can oppose the rollout's advantage, and positive- and negative-advantage rollouts have similar proportions of boosted and suppressed tokens (our Fig. 1).

Here R+(qu)\mathcal R^+(q_u) indexes correct responses to uu's query, TiT_i is the token count of response yiy_i, and sus_u is +1+1 or −1-1 for a correct or incorrect training rollout, B\mathcal B is the update batch, NN its token count, and AuA_u the effective update weight.

We further characterize the same-token, low-confidence coupling structure (Sec. 2) and test the cancellation hypothesis using Monte Carlo estimates of token value (Fig. 5) and interventions on batch composition (Figs. 6–7). These analyses motivate query-preserved balancing and reward-balanced buffering (Sec. 4). We have revised Sec. 2 and the related-work discussion to credit THR's findings more clearly and explain these connections.

[1] Deng et al. Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning.

AuthorsOpenReview ↗

Response to reviewer 4bKS part 2/n

W2. Evaluation beyond Qwen Series Model

The experiment is only on the Qwen model; more model families like Llama or Numotron should be included.

We extend our evaluation to two additional model families: Llama-3.2-3B-Instruct and OLMo-3-1025-7B. We also attempted RL training with suggested Nemotron model, but encountered training collapse within 20 steps: similar infra issues have also been reported(verl #7146 and verl #4671).

The results are shown below.

ModelMethodAIME24AIME25AMCOlympiadBenchMinervaMATH-500Avg.
Llama-3.2-3B-InstructBaseline12.500.0031.1018.0715.8149.2021.11
Llama-3.2-3B-InstructQB11.880.7333.7018.5220.2253.2023.04
Llama-3.2-3B-InstructRB12.600.8333.6219.5622.4355.0024.01
Llama-3.2-3B-InstructQB + RB12.810.5234.1120.1522.4355.0024.17
OLMo-3-1025-7BBaseline34.7925.9464.8049.6336.7685.8049.62
OLMo-3-1025-7BQB33.0226.4668.1950.5237.5086.4050.35
OLMo-3-1025-7BRB34.4827.9267.5551.8541.1887.6051.76
OLMo-3-1025-7BQB + RB36.0429.0666.2350.9642.6588.6052.26

Both QB and RB individually improve the performance over the baseline for each model, and their combination achieves the best average score: 21.11 → 24.17 on Llama and 49.62 → 52.26 on OLMo. These additional results provide evidence that the benefits of our batching strategies extend beyond the Qwen family. We have added them to Table 2 in the main text of the revised manuscript.

W3. Computational overhead and sample utilization

Preserving complete query groups may restrict token-budget balancing .... Reward-balanced buffering can additionally reduce sample utilization and increase update latency.

QB overhead. We agree that keeping query groups intact can make it harder to balance token counts across data-parallel workers. We thus measured actor-update time and total iteration time over the 100 iterations. QB increased actor-update time a bit, but rollout generation still accounted for most of each iteration, and total iteration time remained close to the respective baseline.

ModelMethodRollout (s)Update (s)Iteration (s)Update Share (%)
Llama-3.2-3B-InstructBaseline26.436.3740.9615.55
Llama-3.2-3B-InstructQB26.056.8440.7416.79
OLMo-3-1025-7BBaseline138.7745.02207.1721.73
OLMo-3-1025-7BQB139.2448.66210.4423.12

RB overhead. We agree that RB can reduce sample utilization and delay policy updates. We ran two additional experiments to quantify the trade-off. In the first, we compared our RL baseline with and without RB at comparable rollout budgets on Llama-3.2-3B-Instruct. Under similar sampling budget, RB achieves a higher performance. Similar pattern could be found with Qwen2.5-Math-7B models (Sec. 4, Fig. 9(a)).

MethodSamplesMATHMinervaOlympiadBenchAIME24AMCAIME25Average
Baseline204,80051.8018.3818.2212.5026.880.1021.32
RB201,72853.2017.6519.7012.6027.940.4221.92
Baseline378,88050.6018.0115.4110.7328.390.1020.54
RB378,88053.8019.1219.5612.2928.650.9422.39
Baseline532,48051.2019.1218.2213.2330.720.6322.19
RB532,48054.0021.6920.7412.8129.930.1023.21

On Qwen2.5-Math-7B with τ=0.4\tau=0.4, RB used 307,200 of 445,440 generated rollouts for updates over the first 300 steps (68.97%). Updates were deferred less frequently after first 75 steps. Utilization rose from 50.25% in steps 1–100 to 76.92% in steps 101–200 and 94.34% in steps 201–300 (Sec. 4, Fig. 9(b)). Furthermore, one could tweak the threshold τ\tau to control how balanced the buffered batch must be before an update. Lowering τ\tau relaxes this requirement and can reduce sampling overhead.

AuthorsOpenReview ↗

Response to reviewer 4bKS part 3/n

Q1. Off-policy rollouts and policy staleness

The contrast is limited in on-policy learning; however, when efficiency is required in RL, often the rollouts can be off-policy in practice due to efficiency requirements, which can lead to low, how the off-policy effect could be studied to further extend the view of the paper.

Thank you for raising this question. We would like to highlight that our submission already considers policy staleness in RL training to some extent from the following perspectives:

(1) Motivating observation. Token flipping (Fig. 1) is observed under our practical GRPO setup, where 128 prompts with 8 rollouts each are split into 4 mini-batches. The policy is updated after each mini-batch, so later updates use rollouts generated by an earlier policy. (2) Theoretical considerations. In Sec. 4, we discuss how splitting query groups across sequential mini-batch updates can disrupt cancellation, as their rollouts contribute under different policy parameters. This motivates QB to preserve complete query groups within each update. (3) Empirical validation. Our QB/RB experiments use four gradient updates per rollout batch, with importance sampling and clipping (Appendix B). We agree that studying how larger policy lags affect token flipping and cancellation, especially in asynchronous RL, would be an interesting direction for future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their careful constructive feedback. We first appreciate their recognition of the following strengths of our work:

  • Token flipping is “a very novel and interesting discovery” (Reviewer t3eP), and the paper is “Clear and easy to follow” (Reviewers 4bKS / iB7T / t3eP).
  • The analyses “provide evidence beyond aggregate benchmark improvements” (Reviewer 4bKS), including “A well-designed, causal-style test of the coupling structure” (Reviewer iB7T) and “theoretical and empirical evidence” (Reviewer t3eP).
  • QB/RB are “simple and easy to implement” (Reviewer 4bKS) and “Simple, mechanism-motivated interventions with large effects in an unstable regime” (Reviewer iB7T), with “consistent and significant improvements on different models and benchmarks” (Reviewer t3eP).

During the rebuttal phase, we ran additional experiments and revised the manuscript to address the concerns summarized below.

Concern raisedOur response and findingsChanges in the revised manuscript
SGD analysis versus AdamW training (t3eP / iB7T)We clarified where SGD is used and how the AdamW analyses in Sec. 3 connect the derivation to training, and repeated the masking experiment with Adam. Masking same-token, low-confidence tokens yields the highest Boost Rate (86.90%) and Mean Boost under full-parameter Adam updates.Sec. 2, Table 1; Appendices D.2–D.3.
Output-layer proxy (t3eP / iB7T)We clarified that the proxy is mainly use to predict where token coupling mostly takes effect. Both LM-head-only and full-parameter masking exp support the predicted same-token, low-confidence pattern. The analysis and interventions in Secs. 3–4 also use the full model.Sec. 2 (Table 1 and Fig. 3(b)), Appendix D.1.
Robustness of the empirical improvements (4bKS / iB7T)We added Llama-3.2-3B-Instruct and OLMo-3-1025-7B, as well as independently repeated baseline, QB, RB, and QB+RB training with three seeds on Qwen2.5-Math-7B. QB+RB achieves the highest overall score when evaluated in multiple runs and with additional model families.Sec. 4, Table 2; Appendix B; Appendix F, Table 7.
RB sampling efficiency and comparison with dynamic sampling (4bKS / t3eP / iB7T)We added DAPO comparisons under matched sampling budgets, measured RB sample utilization, and clarified the threshold-controlled update condition. RB outperforms DAPO at reported sampling budget on Qwen2.5-Math-7B and OLMo. We also show that on Qwen2.5-Math-7B with τ=0.4\tau=0.4, RB utilization rises from 50.25% (steps 1–100) to 94.34% (steps 201–300), which is manageable.Sec. 4, Fig. 9; Appendix B; Appendix F, Fig. 13 and Table 7.

We also clarified the relationship to THR (Secs. 2 and 5, for Reviewer 4bKS), added within-query gradient-overlap measurements (Appendix E.3, for Reviewer t3eP), numerical-precision and threshold-sensitivity checks (Appendix C, Table 5, for Reviewer iB7T), explained why we repeated updates on a fixed batch (Appendix E.2, Fig. 12, for Reviewer t3eP), and reported comparable iteration times with and without QB on Llama and OLMo (W3 response, for Reviewer 4bKS). New and revised manuscript content is marked in blue, and detailed point-by-point responses are provided below.

We again thank all reviewer's effort in reviewing our submission, which provide invaluable suggestion to improve the quality of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.