- Rating
- 3(+1)
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies critic-free reinforcement learning (such as GRPO) from the perspective of token credits. The authors first discover and investigate the token flipping phenomenon in GRPO, where positive and negative rollouts exhibit similar boosted or suppressed token ratios. They then attribute this to token coupling, i.e., gradient interactions between tokens. The authors subsequently propose the cancellation hypothesis, which states that the coupling signals of identical tokens in both positive and negative rollouts cancel each other out within a batch. The authors empirically verify this hypothesis and propose two training strategies, query-preserved mini-batching and reward-balanced batching. Extensive experiments validate the effectiveness of the proposed methods.
Critical strengths
- This paper has a clear and logical structure, and a complete and meticulous narrative logic, making it easy to follow.
- Token flipping is a very novel and interesting discovery that offers a fresh perspective on the token credits issue in critic-free RLs such as GRPO. Figure 1, with its ingenious design, clearly illustrates this phenomenon, demonstrating considerable insight. The authors subsequently validate their analysis with theoretical and empirical evidence.
- The two training strategies proposed by the authors have brought consistent and significant improvements on different models and benchmarks, including some tasks that are difficult for relatively small models.
Critical weaknesses
- The token flipping phenomenon the authors discovered is interesting, but its explanation is not new. For example, the derivation in Section 3.1 is a standard characteristic of all policy gradient methods with baseline subtraction and is not unique. The paper does not show what is specific to GRPO.
- There are some potentially overly strong assumptions and settings that do not conform to reality. For example, the authors claim in Equation 12 that "in the idealized case where the common within-query overlap is approximately a positive constant," which may be inconsistent with reality and lacks necessary empirical evidence. More importantly, the authors use the output-layer as an analysis proxy, and its sign consistency with the full model direction is only about 70%, which leaves roughly 30% disagreement ("random chance" is 50%). In addition, the optimizer (SGD) used in the analysis is inconsistent with the one used for real-world training, although the authors provide some explanation in the appendix. One-step linearization is used to support claims about long-run dynamics, and the empirical setting in the appendix, which repeatedly updates on the same batch of data, is also very strange.
- The proposed method may be very costly. For example, for reward-balanced batching, the cost could be extremely high (e.g., with very low or very high accuracy). Furthermore, discarded samples may contain valuable information that is being wasted. Additionally, dynamic sampling is a very standard practice, somewhat similar to the authors' method, which they do not compare.
Questions for authors
See Weaknesses.
6 comments
Response to reviewer t3eP part 2/n
W2.3. SGD versus AdamW
In addition, the optimizer (SGD) used in the analysis is inconsistent with the one used for real-world training, although the authors provide some explanation in the appendix.
To answer reviewer's question, we clarify the scope of the SGD simplification and distinguish it from our empirical analyses and training experiments with AdamW.
SGD is mainly used for the derivation and the original causal verification. Specifically, we use SGD for the coupling derivation in Sec. 2 and the original experiments in Table 1 and Fig. 3. The token-flipping observations (Fig. 1), the empirical cancellation analyses (Figs. 4–7), and the QB/RB training experiments in Sec. 4 (Fig. 8 and Table 2) all are based on AdamW. In particular, Sec. 3 tests the cancellation hypothesis empirically under AdamW and motivates the batching interventions evaluated in Sec. 4, connecting the simplified analysis to practical training. Prior work has similarly considered SGD to analyze LLM training dynamics [1–4] and demonstrated its practical use in LLM-RL [5, 6].
To further bridge the gap, we have repeated the Table 1 experiment with Adam on Qwen3-1.7B-Base, using a batch from the 20th training step and retaining the optimizer's accumulated momentum. We evaluate 500 target tokens, masking 5 candidate tokens per target, under both LM-head-only and full-parameter updates:
| Same token | Low confidence | LM-head: Boost Rate (%) | LM-head: Mean Boost | Full: Boost Rate (%) | Full: Mean Boost |
|---|---|---|---|---|---|
| — | — | 46.00 | +0.0000 | 53.30 | +0.0023 |
| — | ✓ | 80.30 | +0.0035 | 75.90 | +0.0402 |
| ✓ | — | 56.20 | -0.0003 | 50.40 | +0.0100 |
| ✓ | ✓ | 95.60 | +0.0115 | 86.90 | +0.1414 |
The same-token, low-confidence selection yields the highest Boost Rate and Mean Boost under both settings. These results provide direct evidence that the coupling structure identified in Sec. 2 also persists under Adam updates.
We also want to clarify why gradient cancellation could generally make sense under AdamW. Because contributions from tokens with positive and negative advantages are summed to form the batch gradient. Opposing components can therefore cancel before AdamW update its first- and second-moment estimates.
[1] Learning Dynamics of LLM Finetuning, ICLR 2025
[2] Deng et al. Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning. 2025. arXiv:2510.03669.
[3] On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization, NeurIPS 2025
[4] On GRPO collapse in search-r1: The lazy likelihood-displacement death spiral, ICML 2026
[5] Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs, ICML 2026
[6] Score Centering Stabilizes Off-policy Reinforcement Learning
W2.4. One-step analysis and long-run dynamics
One-step linearization is used to support claims about long-run dynamics, …
The derivation in Sec. 2 indeed describe a single update and motivates the cancellation hypothesis. We then test its implications for token-level credit assignment empirically under longer training. Specifically, our Monte Carlo token-value analysis (Sec. 3, Fig. 5(a)) shows that boosted tokens have higher estimated value than suppressed ibes, regardless of their rollout's reward sign. This training observation provides empirical support for the token-level credit-assignment interpretation of our cancellation hypothesis.
W2.5. Repeated updates on the same batch
… and the empirical setting in the appendix, which repeatedly updates on the same batch of data, is also very strange.
Thank you for raising this question. In the experiment in Appendix E.2 (Fig. 12), we deliberately keep the batch fixed in order to examine whether the value gap we observed in Fig. 5(a) becomes more pronounced with repeated updates. Fig. 12 shows that the mean value gap between boosted and suppressed tokens increases overall with repeated updates, providing further empirical support for our cancellation hypothesis. We have clarified this motivation in Appendix E.2.
Response to reviewer t3eP part 3/n
W3. RB sampling cost and comparison with dynamic sampling
The proposed method may be very costly. For example, for reward-balanced batching, the cost could be extremely high (e.g., with very low or very high accuracy). Furthermore, discarded samples may contain valuable information that is being wasted. Additionally, dynamic sampling is a very standard practice, somewhat similar to the authors' method, which they do not compare.
Thank you for raising this concern. We agree that RB can introduce additional sampling overhead per training step, especially when rewards are very sparse or saturated. However, we want to highlight that this overhead is adjustable through the threshold . Lowering relaxes the balancing requirement and can reduce the number of additional rollouts needed, allowing practitioners to adjust accroding to their sampling budget.
Comparison with dynamic sampling. Dynamic sampling proposed to do oversampling on certain queries. We compare RB with DAPO on OLMo-3-1025-7B under matched rollout budgets. Overall is the average score across AIME24, AIME25, AMC, OlympiadBench, and Minerva.
| Rollout budget | DAPO overall (%) | RB overall (%) |
|---|---|---|
| 100,000 | 34.72 | 37.43 |
| 200,000 | 38.41 | 41.78 |
| 250,000 | 39.33 | 42.58 |
| 280,000 | 40.76 | 43.42 |
RB achieves a higher overall score at all four matched rollout budgets. We observe the same advantage on Qwen2.5-Math-7B with :
| Rollout budget | DAPO overall (%) | RB overall (%) |
|---|---|---|
| 100,000 | 35.14 | 36.93 |
| 200,000 | 37.16 | 38.50 |
| 300,000 | 39.00 | 42.35 |
| 400,000 | 39.00 | 43.71 |
RB achieves a higher overall score at all four budgets. The corresponding training curve is included in Sec. 4 (Fig. 9(a)) of the main text. We also compare DAPO, QB, RB, and QB+RB on Qwen2.5-Math-7B across multiple random seeds:
| Method | AIME24 | AIME25 | AMC | OlympiadBench | Minerva | MATH-500 | Overall |
|---|---|---|---|---|---|---|---|
| DAPO | 24.10 ± 1.42 | 12.29 ± 0.75 | 58.35 ± 2.08 | 38.17 ± 1.34 | 30.64 ± 1.29 | 75.40 ± 1.39 | 39.82 ± 0.91 |
| QB | 24.26 ± 3.41 | 11.52 ± 0.65 | 60.93 ± 0.56 | 40.27 ± 2.10 | 33.93 ± 2.46 | 78.60 ± 2.60 | 41.59 ± 0.72 |
| RB | 25.55 ± 1.88 | 12.11 ± 1.03 | 61.54 ± 1.37 | 38.78 ± 1.05 | 35.28 ± 1.67 | 77.13 ± 0.61 | 41.73 ± 0.58 |
| QB + RB | 28.12 ± 0.73 | 15.23 ± 0.85 | 62.62 ± 1.34 | 42.49 ± 2.29 | 34.55 ± 1.31 | 81.40 ± 1.39 | 44.07 ± 0.83 |
Both QB and RB achieve higher mean scores than DAPO on five of the six benchmarks, while QB+RB improves all six.
Sample utilization. During the first 300 training steps, RB uses 307,200 of 445,440 generated rollouts for policy updates, giving a cumulative utilization of 68.97%. Utilization over steps 1–100, 101–200, and 201–300 is 50.25%, 76.92%, and 94.34%, respectively. After warm-up ends at around step 75, update deferrals become less frequent and sample utilization improves overall. Sec. 4 (Fig. 9(b)) compares the utilization curves of RB and DAPO dynamic sampling. As a possible extension, asynchronous training could retain positive and negative rollouts in separate buffers for balancing across batches, further reducing sample waste.
We have added the Qwen2.5-Math-7B sampling-budget and sample-utilization curves to Sec. 4 of the main text (Fig. 9). The OLMo budget comparison (Fig. 13) and multi-seed results (Table 7) are provided in Appendix F.
Finally, we would like to highlight that QB and RB are simple follow-up interventions motivated by our proposed cancellation hypothesis. They are intended primarily to test whether preserving or strengthening cancellation improves training, and their empirical benefits provide further support for this hypothesis.
[1] Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Paper.
Response to reviewer t3eP part 1/n
W1. Scope beyond GRPO
Thank you for raising this point. We agree that the covariance identity in the original Sec. 3.1 follows from centering the advantages, and we have moved the derivation to Appendix E.3. Furthermore, we do think that Cancellation can arise beyond GRPO when rollout gradients with opposite weights share directions. We observe a similar pattern with REINFORCE and a global baseline: template tokens account for 32.4% of the log-probability boost with REINFORCE, compared with 51.2% and 52.2% in the positive-only and negative-only training, respectively (Table 6 in our revised manuscript).
Our contribution could be summarized as characterization of how token coupling shapes implicit credit assignment, including (1) A token’s log-probability change depends on the contributions of other tokens in the batch, so the direction of its update can oppose its own rollout’s advantage. (2) Our analysis predicts strong coupling between identical, low-confidence tokens, which we validate empirically (Table 1 and Fig. 3(b)). (3) This motivates our cancellation hypothesis: opposing signals can cancel along directions shared by successful and unsuccessful rollouts, leaving more selective updates. Consistent with this interpretation, boosted tokens have higher Monte Carlo estimated value than suppressed tokens for both rollout advantage signs (Fig. 5(a)).
We chose GRPO as our baseline because it is widely used for critic-free RL method.
W2. Assumptions and analytical settings
W2.1. Within-query gradient overlap
Thank you for raising this concern. Our intuition is that correct and incorrect rollouts for the same query can draw on similar knowledge and share intermediate reasoning steps, which motivates our expectation of positive overlap between their response-level gradients.
To examine this intuition empirically, we measured rollout-gradient cosine similarity for 20 queries with 8 rollouts each (160 rollouts in total). We find that 94.82% of within-query pairs have positive gradient cosine similarity. The mean gradient cosine similarity is 0.191 within queries, compared with 0.131 across queries.
These measurements show that within-query gradient overlap is predominantly positive. We use a common positive constant cq as an idealized approximation to illustrate the cancellation mechanism. The derivation, originally in Sec. 3.1 of the submission, now appears in Appendix E.3 of the revised manuscript, where we report these measurements in a footnote.
W2.2. Validity of the output-layer proxy
Thank you for raising this concern. We use the output-layer proxy in Sec. 2 mainly to identify where coupling is strongest. The proxy predicts that coupling concentrates on identical, low-confidence token pairs (Eq. 10).
More importantly, the pattern predicted under such assumption are further validated with full parameter updates. Table 1 shows that masking tokens that meet both conditions yields similar Boost Rates (74.12% and 75.00%) and Mean Boosts (+0.0301 and +0.0306) under both LM-head-only and full-parameter updates, respectively. Figure 3(b) also shows that Boost Rate increases with predicted coupling strength in both settings. The new results with Adam in Table 1 show the same pattern. Masking identical, low-confidence tokens yields the highest Boost Rate and Mean Boost in both settings. In addition, all other analyses and interventions in Secs. 3–4 use the full model.
Note that related work also uses restricted gradient proxies for token-level analysis. OTB uses logit gradients to approximate token gradient norms [1], and DelTA uses LM-head gradients to estimate token coefficients [2].
[1] Li et al. The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL. ICML 2026. Paper.
[2] Zhang et al. DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards. 2026. arXiv:2605.21467.