acceptodds
Under review as a conference paper at ICLR 2027

DelTA: Discriminative Token Credit Assignment for Reinforcement Learning with Verifiable Rewards

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a central technique for improving the reasoning capabilities of large language models. Despite its effectiveness, how response-level rewards translate into token-level probability changes remains poorly understood. We introduce a discriminator view of RLVR updates, showing that the policy-gradient update direction implicitly acts as a linear discriminator over token-gradient vectors and thereby determines which token probabilities are increased or decreased during learning. Under standard sequence-level RLVR, this discriminator is constructed from positive- and negative-side centroids formed by advantage-weighted averaging of token-gradient vectors. However, such centroid construction can be dominated by shared high-frequency patterns, such as formatting tokens, diluting sparse yet discriminative directions that better distinguish high-reward responses from low-reward ones. To address this limitation, we propose **DelTA**, a discriminative token credit assignment method that estimates token coefficients to amplify side-specific token-gradient directions and downweight shared or weakly discriminative ones. These coefficients reweight a self-normalized RLVR surrogate, making the effective side-wise centroids more contrastive and thereby reshaping the RLVR update direction. On seven mathematical benchmarks, DelTA outperforms the strongest same-scale baselines by 3.26 and 2.62 average points on Qwen3-8B-Base and Qwen3-14B-Base, respectively. Additional results on code generation, a different backbone, and out-of-domain evaluations further demonstrate the generalization ability of DelTA.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning with Verifiable Rewards, LLM reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_G2eYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DelTA, a token-level credit assignment method for RLVR. It interprets the local policy-gradient update as an implicit linear discriminator over token-gradient vectors, refines positive- and negative-side centroids, and uses distance-based token weights in a DAPO-style objective. An LM-head gradient proxy reduces computational cost. Experiments on seven math benchmarks report average gains of 3.26 and 2.62 points over the strongest baselines for Qwen3-8B and 14B, respectively. The paper also includes code-generation, out-of-domain, and alternative-backbone evaluations.

Critical strengths
  • The method shows consistent improvements across seven math benchmarks at both model scales, with additional positive results on code generation, OOD tasks, and Olmo3-7B. Appendix J.12 also shows that DelTA retains its advantage over SAPO under approximately matched wall-clock budgets, strengthening the empirical evidence.

  • The paper provides several meaningful ablations, including within-side-only weighting, random weights, and the main coefficient-design choices. Appendix J.8 evaluates the agreement between the LM-head gradient proxy and full-parameter gradients, while Appendix J.10 shows that token-level weighting performs better than trajectory-level weighting. These results provide useful evidence that the proposed token weights help optimization, although the underlying mechanism is not yet fully established.

Critical weaknesses
  1. The proposed mechanism and its advantage over simpler weighting methods are not fully established. The paper argues that shared token-gradient patterns weaken discrimination between positive and negative tokens. However, the actual update depends on the mass-weighted centroid difference (Eq. 3), while Table 15 only measures centroid separation. Although Table 3 shows that refinement improves DelTA (23.27 vs. 19.97), it remains unclear whether simpler gradient-norm or gradient-alignment weighting could achieve similar results. A comparison under matched settings, together with a clearer distinction from ResRL [1], would better establish the proposed mechanism and its novelty.

  2. Checkpoint selection and statistical evidence do not fully establish training robustness. Checkpoints are selected using AIME25, which is also part of the final evaluation (Appendix G). Appendix J.13 provides an alternative AIME24 selection check, but only for the reported 8B DAPO–DelTA comparison, not the strongest baselines. Moreover, Appendix I reports one training run per method, while the significance tests use 16 evaluation samples. These tests capture generation variability, not training-seed variability. Independent checkpoint selection and multiple training seeds would make the reported gains more convincing.

Questions for authors
  1. Can the authors provide direct evidence that shared gradient patterns interfere with the actual policy update, beyond the centroid similarity analysis in Table 15?

  2. Could the authors compare DelTA with simpler gradient-norm and signed gradient-alignment weighting under the same coefficient bounds, normalization, and training budget? How does the proposed mechanism differ from ResRL [1]?

  3. Can the authors report the DelTA-versus-SAPO/FIPO comparisons using independently selected checkpoints or fixed training steps, together with training-seed variation for the main results?

[1] Lin, Zi-Han et al. “ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning.” ArXiv abs/2605.00380 (2026)

4 comments
AuthorsOpenReview ↗

Author Response (1/2)

We thank you for the constructive feedback. We provide clarifications and additional experimental results below in response to your comments. We will ensure that the experimental results and expanded discussions are incorporated into our revised manuscript. Unless otherwise specified, additional experiments use Qwen3-8B-Base and follow the four-benchmark analysis protocol in Section 5. Avg. denotes accuracy weighted by the number of questions in each benchmark.

W1&Q2: Comparison with simpler baselines and ResRL


We conducted additional experiments with ResRL [1] and these simpler credit-assignment baselines:

  • Grad Norm: Assigns higher weights to tokens with larger gradient-proxy norms.

  • Signed Gradient Alignment: Assigns higher weights to tokens whose advantage-signed gradient-proxy contributions are more aligned with the aggregate contribution of the other tokens.

To ensure a fair comparison, all methods use the same backbone, training data, rollout budget, optimizer, and number of training steps. For the two gradient-based baselines, we additionally use the same gradient proxy, coefficient bounds, and normalization scheme as DelTA, so that the comparison isolates the effect of the credit-assignment criterion.

AIME25AIME26HMMT25HMMT26Avg.
ResRL23.9623.9614.1716.8619.67
Grad Norm23.7523.1311.6716.1018.60
Signed Gradient Alignment24.1723.3312.0816.2918.90
DelTA26.4628.1218.5420.2723.27

DelTA consistently outperforms these baselines across all four benchmarks.

Furthermore, ResRL and DelTA differ in their weighting criteria and treatment of shared components. ResRL computes negative-advantage token weights from projection residuals relative to a subspace estimated from positive-advantage token representations, reducing penalties on negative-advantage tokens that overlap with positive-advantage responses. DelTA uses relative distances to the refined positive- and negative-side centroids of token-gradient proxies to assign weights on both advantage sides. Its objective is to emphasize outcome-discriminative gradient patterns in the policy update. Thus, ResRL protects shared components from excessive negative penalties, while DelTA emphasizes gradients that distinguish positive- from negative-advantage responses.

References:

[1] ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning.

W2-1&Q3-1: Checkpoint selection


To address the checkpoint selection concern, we additionally evaluate DelTA and its strongest baselines using checkpoints selected based on AIME24, which is independent of the evaluation benchmarks reported below.

For Qwen3-8B, we compare DelTA against SAPO using AIME24-selected checkpoints. For Qwen3-14B, we conduct the same comparison between DelTA and FIPO.

Qwen3-8BAIME25AIME26HMMT25HMMT26Avg.
SAPO24.5826.0416.2517.4220.98
DelTA26.4628.1218.5420.2723.27
Qwen3-14BAIME25AIME26HMMT25HMMT26Avg.
FIPO34.5842.9232.0824.8133.38
DelTA37.9245.2132.9226.8935.52

For DelTA, AIME24 selects the same checkpoint as the original selection criterion. With checkpoints selected using AIME24, DelTA outperforms SAPO on Qwen3-8B and FIPO on Qwen3-14B by 2.29 and 2.14 percentage points in weighted average accuracy. These results show that DelTA’s advantage persists under checkpoint selection independent of the reported evaluation benchmarks.

W2-2&Q3-2: Results across independent training runs


To assess the robustness of DelTA to training randomness, we conducted three independent training runs for both DelTA and SAPO using different random seeds while keeping all other training configurations fixed. For each run, we selected the checkpoint based on AIME24 performance and evaluated it on the four benchmarks reported below.

SAPOAIME25AIME26HMMT25HMMT26Avg.
Run124.5826.0416.2517.4220.98
Run225.0026.8815.8317.2321.14
Run324.1725.8316.0418.9421.19
DelTAAIME25AIME26HMMT25HMMT26Avg.
Run126.4628.1218.5420.2723.27
Run227.2928.7517.7120.0823.38
Run326.8827.9219.3821.0223.73

Across the three runs, DelTA achieves a weighted average accuracy of 23.46 ± 0.24 (mean ± standard deviation), consistently outperforming SAPO. Notably, DelTA outperforms SAPO in all three runs, demonstrating that its performance gains are reproducible across different random seeds rather than dependent on a particular training run.

AuthorsOpenReview ↗

Author Response (2/2)

Q1: Direct decomposition of shared-pattern contributions


We conducted a full-parameter gradient-decomposition experiment to directly examine how shared-pattern tokens contribute to the RLVR policy gradient. All objectives are evaluated at the same policy on the same sampled responses.

We randomly sampled 256 questions from the training set and generated 16 responses per question. We identified shared patterns independently of DelTA’s coefficients. We treated each contiguous 3-gram as a candidate pattern. For each question, we computed the occurrence rate of each 3-gram among positive- and negative-advantage responses, where the occurrence rate is defined as the fraction of responses containing that 3-gram. A 3-gram was identified as a shared pattern if it appeared in at least 30% of responses on both sides, with an absolute difference between the two occurrence rates no greater than 0.2. All token positions covered by at least one identified shared pattern were labeled as shared, while all other positions were labeled as remaining.

Let G+,SG_{+,S} and G−,SG_{-,S} denote the aggregated gradients of shared-pattern tokens on the positive- and negative-advantage sides. We aggregate token-level gradients using the absolute values of their corresponding objective coefficients. We retain the original coefficient magnitudes without normalizing the positive- and negative-side aggregates, thereby preserving their actual, potentially unequal gradient masses.

Their signed net contribution is gS=G+,S−G−,S.g_S = G_{+,S} - G_{-,S}. Let gRg_R denote the net gradient contribution from all remaining tokens. The full gradient can then be decomposed as
g=gS+gR.g = g_S + g_R. We report the following metrics:

  1. Directional alignment: We report cos⁡(G+,S,G−,S)\cos(G_{+,S},G_{-,S}), measuring the directional alignment between the positive- and negative-side shared-pattern gradients, and cos⁡(gS,gR)\cos(g_S,g_R), measuring the alignment between the net shared-pattern contribution and the remaining-token contribution.
  2. Matched random-token control: For each response, we uniformly sample the same number of token positions as identified shared-token positions. Let G+randG_{+}^{\mathrm{rand}} and G−randG_{-}^{\mathrm{rand}} denote the aggregated gradients of the sampled tokens on the positive- and negative-advantage sides. We report their cross-side cosine similarity, cos⁡(G+rand,G−rand)\cos(G_{+}^{\mathrm{rand}},G_{-}^{\mathrm{rand}}). We also compute the gradient remaining after removal of the sampled contributions, gdelrand=g−(G+rand−G−rand),g_{\mathrm{del}}^{\mathrm{rand}} = g-\left(G_{+}^{\mathrm{rand}}-G_{-}^{\mathrm{rand}}\right), and report the random-deletion angle, θrand=∠(g,gdelrand).\theta_{\mathrm{rand}} = \angle(g,g_{\mathrm{del}}^{\mathrm{rand}}).
  3. Gradient norm ratios: We report ∥G+,S∥2/∥G−,S∥2\lVert G_{+,S} \rVert_2/\lVert G_{-,S} \rVert_2 to compare the magnitudes of the positive- and negative-side shared-pattern aggregates. We additionally report ∥gS∥2/∥gR∥2\lVert g_S \rVert_2/\lVert g_R \rVert_2 and ∥gS∥2/∥g∥2\lVert g_S \rVert_2/\lVert g \rVert_2, measuring the magnitude of the non-cancelled shared-pattern contribution relative to the remaining-token contribution and the full gradient.
  4. Angular deviation: We measure how much removing the shared-pattern contribution changes the full gradient direction: θS=∠(g,gR)=arccos⁡(g⊤gR∥g∥2∥gR∥2).\theta_S = \angle(g,g_R) = \arccos\left( \frac{g^\top g_R}{\lVert g\rVert_2\lVert g_R\rVert_2} \right).
DAPODelTA
cos⁡(G+,S,G−,S)\cos(G_{+,S}, G_{-,S})0.990.97
cos⁡(gS,gR)\cos(g_S,g_R)-0.77-0.66
cos⁡(G+rand,G−rand)\cos(G_+^{\mathrm{rand}},G_-^{\mathrm{rand}})0.08-0.59
∥G+,S∥2/∥G−,S∥2\lVert G_{+,S}\rVert_2/ \lVert G_{-,S}\rVert_20.820.68
∥gS∥2/∥gR∥2\lVert g_S\rVert_2/\lVert g_R\rVert_20.800.59
∥gS∥2/∥g∥2\lVert g_S\rVert_2/\lVert g\rVert_21.260.78
∠(g,gR)\angle(g, g_R)53.3°35.8°
∠(g,gdelrand)\angle(g,g_{\mathrm{del}}^{\mathrm{rand}})40.7°20.2°

The positive- and negative-side shared gradients are highly aligned, whereas the count-matched random-token aggregates exhibit substantially lower cross-side alignment. Despite partial cancellation, the shared-pattern residual remains substantial relative to the remaining-token contribution and is negatively aligned with it. Removing shared-pattern contributions changes the gradient direction by 53.3° for DAPO and 35.8° for DelTA, compared with changes of 40.7° and 20.2° under count-matched random removal. DelTA exhibits a lower shared-to-remaining gradient norm ratio and a smaller angular deviation. These results provide direct evidence that independently identified shared-pattern tokens have a non-negligible net contribution and substantially affect the gradient direction.

Reviewer_G2eYOpenReview ↗

Official Review of Submission26076 by Reviewer G2eY

Thank you for the detailed rebuttal and the additional experiments. I appreciate the authors' efforts to address my concerns.

The additional baseline comparisons, gradient analysis, and independent training runs substantially address my concerns and strengthen the evidence for DelTA's effectiveness and robustness.

However, one concern remains regarding the interpretation of the shared-gradient analysis and its role in explaining DelTA's improvements. My original concern was whether shared gradient patterns actually weaken the induced discriminator, rather than simply making the positive- and negative-side centroids more similar.

The new decomposition shows that shared-pattern gradients make a substantial contribution to the net gradient and that DelTA reduces their relative influence in this analysis. This is useful evidence. However, a large contribution, even when negatively aligned with the remaining gradient, does not necessarily mean that it makes the policy update less effective. The remaining gradient is not necessarily a better update direction. More specifically, the decomposition does not directly show that shared-pattern gradients reduce the induced discriminator's ability to distinguish positive- from negative-advantage token gradients, or that reducing their influence explains DelTA's performance gains.

I acknowledge that the within-side-only ablation, refinement ablation, and token-selection experiments provide meaningful support for DelTA's weighting strategy. Together with the new decomposition, these results strengthen the proposed explanation, but do not fully establish the specific role of shared-gradient interference. I therefore consider this concern partially addressed and encourage the authors to make this distinction clear when discussing the proposed mechanism.

Overall, despite the remaining concern about the interpretation of the shared-gradient analysis, the rebuttal has substantially strengthened the paper, and I remain positive and maintain my original score.

We thank the reviewer for this important clarification. We agree that a substantial shared-gradient contribution, even when negatively aligned with the remaining gradient, does not necessarily imply that it harms policy optimization.

To investigate this question more directly, we conducted an additional training-time intervention that explicitly amplifies shared-pattern contributions during RL optimization.

Specifically, we identify shared-pattern tokens using the same 3-gram criterion described in our previous response. We then modify the DAPO objective by assigning 2× weight to shared-pattern tokens, while keeping the weights of all remaining tokens at 1×. The weighted loss is normalized by the sum of token weights to avoid trivially increasing the overall loss scale. All other training configurations remain unchanged.

Under the local policy-gradient formulation, this intervention increases the relative contribution of shared-pattern gradients, corresponding to a direction proportional to

gShared×2=gR+2gS,g_{\mathrm{Shared}\times 2}=g_R+2g_S,

compared with the original DAPO direction gDAPO=gR+gSg_{\mathrm{DAPO}}=g_R+g_S.

Due to time and computational resource constraints, we conducted this controlled intervention for 80 training steps and compared its performance with standard DAPO at the same training step, keeping all other training configurations identical.

The results are summarized below.

MethodAIME25AIME26HMMT25HMMT26Avg.
DAPO21.8822.7110.6315.1517.53
DAPO w/ Shared ×220.2120.638.7514.2015.90

Amplifying shared-pattern contributions reduces the weighted average accuracy from 17.53 to 15.90, a decrease of 1.63 percentage points compared with standard DAPO.

Importantly, this experiment goes beyond characterizing gradient geometry: by directly intervening on shared-pattern contributions during RL training, we find that increasing their relative influence leads to lower downstream performance in this controlled training experiment. This provides intervention-based evidence that excessive emphasis on shared patterns can be detrimental to effective policy optimization, consistent with our shared-gradient interference hypothesis.

While this experiment does not directly measure the induced discriminator's classification performance, it provides evidence concerning the actual optimization consequences of emphasizing shared-pattern gradients, which is the central motivation of our method.

Together with the previous full-parameter gradient decomposition, refinement ablation, and token-selection experiments, these results provide complementary support for our proposed mechanism: shared patterns substantially influence the policy update, excessive emphasis on these patterns can impair training, and DelTA's selective weighting may help mitigate this interference.

We appreciate the reviewer's constructive feedback, which has helped us strengthen both the experimental evidence and the interpretation of our proposed mechanism.

Reviewer_gT6kOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how sequence-level RLVR rewards translate into token-level probability changes. Using a first-order expansion of the DAPO update, it argues that the update acts as an implicit linear discriminator over token-gradient vectors. The discriminator is defined by advantage-weighted positive- and negative-side centroids, which can be dominated by shared, weakly discriminative directions. The authors propose DelTA, which gives each token a bounded, self-normalized coefficient. The coefficient is high when the token’s gradient proxy is closer to its own side's centroid than to the opposite side’s, and it is used to reweight the DAPO surrogate.

DelTA outperforms DAPO, DAPO with forking tokens, SAPO, and FIPO on seven math benchmarks for Qwen3-8B/14B-Base. It also improves DAPO on Olmo3-7B, code generation, GRPO, and out-of-domain benchmarks. The ablations show three things: the opposite-side comparison is necessary; training only on high-coefficient tokens beats random selection, while low-coefficient tokens collapse; and neither random coefficients nor a larger learning rate reproduce the gains.

Critical strengths
  1. A simple, intuitivemethod with consistent gains. DelTA is a plug-in token reweighting of the DAPO surrogate with modest overhead. It outperforms strong token-level and policy-optimization baselines on both Qwen3 backbones. The gains carry over to another model family, code generation, GRPO and out-of-domain benchmarks, and remains effective under matched wall-clock budgets. This is consequential because the method is cheap to adopt, and consistency across settings means that it generalizes well.
  2. Comprehensive experimental setup and insightful ablations. The controls address the obvious alternative explanations. Random coefficients, within-side-only scoring and trajectory-level weighting all fall short. Training only on high-coefficient tokens helps, while low-coefficient tokens collapse training. A larger learning rate does not reproduce the gains. The method is also tested across model families and datasets. Overall, I really like the experimental design and setup, except for some missing baselines in the main table.
Critical weaknesses
  1. The gains are not yet cleanly attributable to the proposed reweighting. Several confounds and missing comparisons leave open whether the improvement comes from discriminative reweighting itself.
  • Possibly weakened baseline. Dynamic sampling is disabled for all methods, and the DAPO baseline shows atypical dynamics in Fig. 2: reward declines, entropy rises steadily, and response length roughly halves. It is unclear whether the gains hold under the standard DAPO recipe.
  • A scale-matched DAPO baseline is missing. RLVR performance can improve simply by scaling the update, so the gains need to be separated from update magnitude. Zero-advantage tokens receive λ_min but still count in the normalizer Z. Self-normalization therefore scales up the nonzero-advantage tokens, increasingly so as zero-advantage groups grow. A DAPO control with its loss rescaled to match DelTA's loss scale is needed.
  • Missing baselines. There is no comparison with entropy-based advantage shaping [1] or probability-based Advantage Reweighting [2].
  1. The proposed mechanism is asserted rather than fully justified. The paper's contribution is framed around this mechanism, so it needs direct support with empirical preliminary experiments.
  • No direct evidence. The claim that shared, high-frequency patterns dominate the centroids rests on intuition. The paper reports neither the distributions of positive- and negative-side token gradients nor how much of each centroid comes from shared versus side-specific directions.
  • Tension with gradient scaling. High-probability tokens have near-zero gradients, as the proxy’s (1 − p) factor shows, so formatting tokens should contribute little to the centroids. [2] find instead that low-probability tokens dominate updates. The high centroid similarity in Table 15 may simply reflect hidden-state anisotropy.
  • Loose connection between theory and proxy implementation. The proxy’s coefficients agree with full-gradient coefficients at only 0.47 Spearman (Table 13).

NOTE: the paper has no limitation section.

References:

[1] Cheng, Daixuan, Shaohan Huang, Xuekai Zhu, Bo Dai, Xin Zhao, Zhenliang Zhang, and Furu Wei. "Reasoning with exploration: An entropy perspective." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 36, pp. 30377-30385. 2026.

[2] Yang, Zhihe, Xufang Luo, Zilong Wang, Dongqi Han, Zhiyuan He, Dongsheng Li, and Yunjian Xu. "Do not let low-probability tokens over-dominate in rl for llms." In International Conference on Learning Representations, vol. 2026, pp. 117072-117101. 2026.

Questions for authors
  1. Why is dynamic sampling disabled? Does DelTA still work with DAPO recipe with dynamic sampling enabled?
  2. Can you provide direct evidence for the shared-pattern hypothesis? Examples: distributions of positive- and negative-side token-gradient vectors; the share of each centroid explained by shared versus side-specific directions; and the correlation between λ and token probability or entropy. Right now it's all hypothetical.
6 comments
AuthorsOpenReview ↗

Author Response (1/3)

We thank you for the constructive feedback. We provide clarifications and additional experimental results below in response to your comments. We will ensure that the experimental results and expanded discussions are incorporated into our revised manuscript. Unless otherwise specified, additional experiments use Qwen3-8B-Base and follow the four-benchmark analysis protocol in Section 5. Avg. denotes accuracy weighted by the number of questions in each benchmark.

Q1&W1-1: Dynamic sampling


We disable dynamic sampling in our original experiments because our primary goal is to isolate the effect of the policy-update objective. As discussed in Section 2, dynamic sampling affects rollout filtering rather than the mathematical form of the per-token surrogate objective.

To directly address the reviewer's question, we additionally train both DAPO and DelTA with dynamic sampling enabled. The results are shown below.

AIME25AIME26HMMT25HMMT26Avg.
DAPO23.3324.1712.0816.8619.05
DAPO w/ Dynamic Sampling24.3824.7912.7116.4819.51
DelTA26.4628.1218.5420.2723.27
DelTA w/ Dynamic Sampling28.3328.3320.4219.5124.03

With dynamic sampling enabled, DelTA outperforms DAPO by 4.52 average points, compared with 4.22 points without dynamic sampling. These results show that DelTA remains effective when dynamic sampling is enabled, and its gains do not depend on disabling this component.

Regarding the training dynamics in Figure 2, we would like to clarify that the observed behavior is not necessarily an anomaly specific to our DAPO implementation. Similar non-monotonic training dynamics have also been reported in Beyond the 80/20 Rule [3]. Specifically, their vanilla DAPO experiments on Qwen3-8B-Base exhibit increasing overall entropy and a late-stage decline in response length. These observations suggest that such training dynamics can naturally arise in RLVR training with Qwen3-8B-Base.

Q2: Relationship to token statistics


To further examine whether DelTA's coefficients merely reflect token probability, predictive entropy, or gradient norm, we compute both Spearman rank correlations and Pearson linear correlations between the token coefficients λ\lambda and these three token-level statistics. We compute these correlations over all valid response tokens in the sampled training batches.

SpearmanPearson
probability-0.1475-0.1653
entropy0.12040.1815
gradient-proxy norm0.14880.1698

All correlations are relatively weak. These results suggest that DelTA’s coefficients do not closely track token probability, predictive entropy, or gradient magnitude through a simple linear or monotonic relationship. In particular, although low-probability tokens may have larger gradient magnitudes, DelTA does not simply assign larger coefficients to tokens with larger gradients.

W1-2&W1-3: Additional baselines


We introduce a normalization-only DAPO control to isolate the effect of DelTA’s batch-dependent normalization. Specifically, let NN denote the total number of valid response tokens and Zλ=∑i,tλi,tZ_\lambda=\sum_{i,t}\lambda_{i,t} denote DelTA's normalization factor. We multiply the original DAPO loss by the same batch-dependent scaling factor N/ZλN/Z_\lambda: L=NZλLDAPO.\mathcal L= \frac{N}{Z_\lambda}\mathcal L_{\mathrm{DAPO}}.

Both NN and ZλZ_\lambda include valid response tokens from zero-advantage groups, as in DelTA. This control therefore tests whether the rescaling induced by DelTA’s normalization alone can reproduce its gains. We also report DAPO with DelTA’s token weights and its original token-count denominator, corresponding to the w/o λ\lambda-norm ablation in Table 3.

We also incorporate two additional related methods: Entropy-Based Advantage Shaping [1] and Probability-based Advantage Reweighting (AR) [2]. We implement both methods within the same DAPO training pipeline, using the same backbone, training data, optimizer, training steps, evaluation settings, and checkpoint-selection protocol.

AIME25AIME26HMMT25HMMT26Avg.
DAPO23.3324.1712.0816.8619.05
DAPO w/ DelTA normalization23.1324.3812.2915.5318.75
DAPO w/ DelTA’s token weights24.3726.2515.8319.3221.39
Entropy-Based Advantage Shaping23.7523.5412.2916.4818.95
AR24.1723.7512.2916.2919.05
DelTA26.4628.1218.5420.2723.27

DAPO with DelTA normalization achieves 18.75, compared with 19.05 for the original DAPO. Applying DelTA’s token weights with the original DAPO denominator improves accuracy to 21.39, and full DelTA further improves it to 23.27. These results show that denominator-induced rescaling alone does not reproduce DelTA’s gains, while discriminative token reweighting remains beneficial under the original DAPO normalization. DelTA also outperforms both additional related methods.

AuthorsOpenReview ↗

Author Response (2/3)

W2-1: Direct decomposition of shared-pattern contributions


We conducted a full-parameter gradient-decomposition experiment to directly examine how shared-pattern tokens contribute to the RLVR policy gradient. All objectives are evaluated at the same policy on the same responses.

We randomly sampled 256 questions from the training set and generated 16 responses per question. We identified shared patterns independently of DelTA’s coefficients. We treated each contiguous 3-gram as a candidate pattern. For each question, we computed the occurrence rate of each 3-gram among positive- and negative-advantage responses, where the occurrence rate is defined as the fraction of responses containing that 3-gram. A 3-gram was identified as a shared pattern if it appeared in at least 30% of responses on both sides, with an absolute difference between the two occurrence rates no greater than 0.2. All token positions covered by at least one identified shared pattern were labeled as shared, while all other positions were labeled as remaining.

Let G+,SG_{+,S} and G−,SG_{-,S} denote the aggregated gradients of shared-pattern tokens on the positive- and negative-advantage sides. We aggregate the token-level gradients using the absolute values of their corresponding objective coefficients. We preserve their actual, potentially unequal gradient masses.

Their signed net contribution is gS=G+,S−G−,S.g_S = G_{+,S} - G_{-,S}. Let gRg_R denote the net gradient contribution from all remaining tokens. The full gradient can then be decomposed as
g=gS+gR.g = g_S + g_R.

We report the following metrics:

  1. Directional alignment: We report cos⁡(G+,S,G−,S)\cos(G_{+,S},G_{-,S}), measuring the directional alignment between the positive- and negative-side shared-pattern gradients, and cos⁡(gS,gR)\cos(g_S,g_R), measuring the alignment between the net shared-pattern contribution and the remaining-token contribution.
  2. Matched random-token control: For each response, we uniformly sample the same number of token positions as identified shared-token positions. Let G+randG_{+}^{\mathrm{rand}} and G−randG_{-}^{\mathrm{rand}} denote the aggregated gradients of the sampled tokens on the positive- and negative-advantage sides. We report their cross-side cosine similarity, cos⁡(G+rand,G−rand)\cos(G_{+}^{\mathrm{rand}},G_{-}^{\mathrm{rand}}). We also compute the gradient remaining after removal of the sampled contributions, gdelrand=g−(G+rand−G−rand),g_{\mathrm{del}}^{\mathrm{rand}} = g-\left(G_{+}^{\mathrm{rand}}-G_{-}^{\mathrm{rand}}\right), and report the random-deletion angle, θrand=∠(g,gdelrand).\theta_{\mathrm{rand}} = \angle(g,g_{\mathrm{del}}^{\mathrm{rand}}).
  3. Gradient norm ratios: We report ∥G+,S∥2/∥G−,S∥2\lVert G_{+,S} \rVert_2/\lVert G_{-,S} \rVert_2 to compare the magnitudes of the positive- and negative-side shared-pattern aggregates. We additionally report ∥gS∥2/∥gR∥2\lVert g_S \rVert_2/\lVert g_R \rVert_2 and ∥gS∥2/∥g∥2\lVert g_S \rVert_2/\lVert g \rVert_2, measuring the magnitude of the non-cancelled shared-pattern contribution relative to the remaining-token contribution and the full gradient.
  4. Within-side norm ratios: Let G+G_+ and G−G_- denote the full positive- and negative-side gradient aggregates. We report ∥G+,S∥2/∥G+∥2\lVert G_{+,S} \rVert_2/\lVert G_+ \rVert_2 and ∥G−,S∥2/∥G−∥2\lVert G_{-,S} \rVert_2/\lVert G_- \rVert_2, measuring the magnitude of shared-pattern contributions relative to the full aggregate on each side. These ratios can exceed one because shared-pattern and remaining-token contributions can partially cancel within the same advantage side.
  5. Angular deviation: We measure how much removing the shared-pattern contribution changes the full gradient direction: θS=∠(g,gR).\theta_S = \angle(g,g_R).
DAPODelTA
cos⁡(G+,S,G−,S)\cos(G_{+,S}, G_{-,S})0.990.97
cos⁡(gS,gR)\cos(g_S,g_R)-0.77-0.66
cos⁡(G+rand,G−rand)\cos(G_+^{\mathrm{rand}},G_-^{\mathrm{rand}})0.08-0.59
∥G+,S∥2/∥G−,S∥2\lVert G_{+,S}\rVert_2/ \lVert G_{-,S}\rVert_20.820.68
∥G+,S∥2/∥G+∥2\lVert G_{+,S} \rVert_2/\lVert G_+ \rVert_27.772.89
∥G−,S∥2/∥G−∥2\lVert G_{-,S} \rVert_2/\lVert G_- \rVert_26.493.54
∥gS∥2/∥gR∥2\lVert g_S\rVert_2/\lVert g_R\rVert_20.800.59
∥gS∥2/∥g∥2\lVert g_S\rVert_2/\lVert g\rVert_21.260.78
∠(g,gR)\angle(g, g_R)53.3°35.8°
∠(g,gdelrand)\angle(g,g_{\mathrm{del}}^{\mathrm{rand}})40.7°20.2°

The positive- and negative-side shared gradients are highly aligned, whereas the random-token aggregates exhibit substantially lower cross-side alignment. Shared-pattern contributions are also substantial relative to the full aggregates on both advantage sides. Despite partial cancellation, the shared-pattern residual remains substantial relative to the remaining-token contribution and is negatively aligned with it. Removing shared-pattern contributions causes a larger directional change than count-matched random removal for both methods. DelTA exhibits a lower shared-to-remaining gradient norm ratio and a smaller angular deviation. Together, these results provide direct evidence that independently identified shared-pattern tokens have a non-negligible net contribution and substantially affect the gradient direction.

AuthorsOpenReview ↗

Author Response (3/3)

W2-2: Clarifying token probability and gradient anisotropy


We first clarify that shared-pattern tokens are not necessarily high-probability tokens. Therefore, the small gradient magnitudes associated with high-probability tokens do not by themselves imply that shared-pattern contributions are negligible. More directly, the full-parameter decomposition in W2-1 shows that the net contribution of the identified shared-pattern tokens is substantial, with ∥gS∥2/∥gR∥2=0.80\lVert g_S\rVert_2/\lVert g_R\rVert_2=0.80 under DAPO.

Moreover, as reported in Q2, DelTA’s coefficients exhibit only weak correlations with token probability, with a Spearman correlation of −0.1475 and a Pearson correlation of −0.1653. These results suggest that DelTA’s coefficients do not simply track token probability.

To further examine whether the observed alignment is merely a consequence of gradient anisotropy, we use randomly sampled token gradients as a reference. As shown in W2-1, their positive- and negative-side aggregated gradients exhibit substantially lower cosine similarities (0.08 for DAPO and -0.59 for DelTA) than those of shared-pattern tokens (0.99 and 0.97). This contrast suggests that the strong alignment is associated with the identified tokens, rather than being a generic property of arbitrary count-matched token subsets.

W2-3: Clarification on the gradient proxy


As discussed in Section 3.2 and Appendix D, our default LM-head proxy is designed as a practical trade-off between gradient fidelity and computational efficiency. Computing full-parameter token gradients for every rollout token is prohibitively expensive at LLM scale. We therefore use the lightweight proxy v^i,t=(1−pi,t)hi,t\hat v_{i,t}=(1-p_{i,t})h_{i,t} only for estimating the stop-gradient coefficients, rather than treating it as an exact reconstruction of the full-parameter gradients. Importantly, the actual weighted RLVR objective is still differentiated over all model parameters.

We further emphasize that the relevant approximation target is preserving the discriminative comparisons used to determine token coefficients, rather than exactly reconstructing every full-parameter gradient vector. As reported in Table 13, although the default proxy achieves a moderate Spearman correlation of 0.470 with full-gradient coefficients, it preserves 88.8% of the top-50% tokens selected using full-gradient coefficients. Moreover, richer gradient representations improve both measures: the top-K hidden-gradient proxy achieves 0.661 Spearman correlation and 0.922 top-50% overlap, while gradients from the last 25% and 50% of model layers achieve 0.885 and 0.900 Spearman correlations with corresponding overlaps of 0.970 and 0.974.

We retain the lightweight LM-head proxy as our default configuration because it offers a favorable trade-off between computational efficiency and coefficient fidelity.

Limitations


We will add a Limitations section to the revised paper, discussing the following aspects:

  1. The approximation introduced by using a lightweight token-gradient proxy instead of full-parameter token gradients, and the associated trade-off between computational efficiency and coefficient fidelity.

  2. The scope of our empirical evaluation, which focuses primarily on mathematical reasoning, with further validation needed in broader RLVR settings such as multi-turn interaction and tool-use tasks.

References:

[1] Reasoning with exploration: An entropy perspective.

[2] Do not let low-probability tokens over-dominate in rl for llms.

[3] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning

Reviewer_FqSCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines how sequence-level RLVR updates change token probabilities and shows that a local, unclipped policy-gradient update acts as an implicit linear discriminator over token-gradient vectors. Motivated by the concern that shared token patterns weaken the contrast between positive- and negative-advantage gradients, DelTA refines side-wise centroids and uses their relative distances to assign bounded token weights in a self-normalized DAPO-style surrogate. In practice, these weights are estimated with a last-layer gradient proxy rather than full-parameter token gradients. Across seven math benchmarks, DelTA raises the mean score over the strongest same-scale baselines from 25.14 to 28.40 on Qwen3-8B-Base and from 37.29 to 39.91 on Qwen3-14B-Base.

Critical strengths
  1. The good point of this paper is to propose to connect token gradients to local probability changes and lead to a specific, bounded weighting rule. The within-side-only ablation, token-selection experiment, and component ablations support the value of contrasting the two advantage sides.

  2. The implementation and evaluations metrics parts are detailed, and the baseline. The main results on a Table 1 shows gains on 7 math benchmark at both model scales.

Critical weaknesses
  1. Shared gradient components can cancel between the positive and negative aggregates. Lower centroid cosine similarity alone does not establish that shared components impair the net update or that reducing them explains the performance gains. A direct decomposition of the net update before and after reweighting would strengthen this claim.

  2. Successful responses can contain unhelpful tokens, while unsuccessful responses can contain useful reasoning steps. DelTA weights tokens by their association with the two advantage sides, and the top-50% experiment demonstrates training utility rather than causal contributions to correctness. More direct intervention-based evidence would strengthen the credit-assignment interpretation.

Questions for authors
  1. Can you directly measure how much the shared/background token directions contribute to the DAPO update before and after DelTA weighting, including the effect of unequal positive/negative masses?

  2. What is the sensitivity of DelTA under different rollouts?

  3. Could you please take more related baselines in to comparison?

3 comments
AuthorsOpenReview ↗

Author Response (1/2)

We thank you for the constructive feedback. We provide clarifications and additional experimental results below in response to your comments. We will ensure that the results and discussions are incorporated into our revised manuscript. Unless otherwise specified, additional experiments use Qwen3-8B-Base and follow the four-benchmark analysis protocol in Section 5. Avg. denotes accuracy weighted by the number of questions in each benchmark.

Q1&W1: Direct decomposition of shared-pattern contributions


We conducted a full-parameter gradient-decomposition experiment to directly examine how shared-pattern tokens contribute to the RLVR policy gradient. All objectives are evaluated at the same policy on the same sampled responses.

We randomly sampled 256 questions from the training set and generated 16 responses per question. We identified shared patterns independently of DelTA’s coefficients. We treated each contiguous 3-gram as a candidate pattern. For each question, we computed the occurrence rate of each 3-gram among positive- and negative-advantage responses, where the occurrence rate is defined as the fraction of responses containing that 3-gram. A 3-gram was identified as a shared pattern if it appeared in at least 30% of responses on both sides, with an absolute difference between the two occurrence rates no greater than 0.2. All token positions covered by at least one identified shared pattern were labeled as shared, while all other positions were labeled as remaining.

Let G+,SG_{+,S} and G−,SG_{-,S} denote the aggregated gradients of shared-pattern tokens on the positive- and negative-advantage sides. We aggregate token-level gradients using the absolute values of their corresponding objective coefficients. We retain the original coefficient magnitudes without normalizing the positive- and negative-side aggregates, thereby preserving their actual, potentially unequal gradient masses.

Their signed net contribution is gS=G+,S−G−,S.g_S = G_{+,S} - G_{-,S}. Let gRg_R denote the net gradient contribution from all remaining tokens. The full gradient can then be decomposed as
g=gS+gR.g = g_S + g_R. We report the following metrics:

  1. Directional alignment: We report cos⁡(G+,S,G−,S)\cos(G_{+,S},G_{-,S}), measuring the directional alignment between the positive- and negative-side shared-pattern gradients, and cos⁡(gS,gR)\cos(g_S,g_R), measuring the alignment between the net shared-pattern contribution and the remaining-token contribution.
  2. Matched random-token control: For each response, we uniformly sample the same number of token positions as identified shared-token positions. Let G+randG_{+}^{\mathrm{rand}} and G−randG_{-}^{\mathrm{rand}} denote the aggregated gradients of the sampled tokens on the positive- and negative-advantage sides. We report their cross-side cosine similarity, cos⁡(G+rand,G−rand)\cos(G_{+}^{\mathrm{rand}},G_{-}^{\mathrm{rand}}). We also compute the gradient remaining after removal of the sampled contributions, gdelrand=g−(G+rand−G−rand),g_{\mathrm{del}}^{\mathrm{rand}} = g-\left(G_{+}^{\mathrm{rand}}-G_{-}^{\mathrm{rand}}\right), and report the random-deletion angle, θrand=∠(g,gdelrand).\theta_{\mathrm{rand}} = \angle(g,g_{\mathrm{del}}^{\mathrm{rand}}).
  3. Gradient norm ratios: We report ∥G+,S∥2/∥G−,S∥2\lVert G_{+,S} \rVert_2/\lVert G_{-,S} \rVert_2 to compare the magnitudes of the positive- and negative-side shared-pattern aggregates. We additionally report ∥gS∥2/∥gR∥2\lVert g_S \rVert_2/\lVert g_R \rVert_2 and ∥gS∥2/∥g∥2\lVert g_S \rVert_2/\lVert g \rVert_2, measuring the magnitude of the non-cancelled shared-pattern contribution relative to the remaining-token contribution and the full gradient.
  4. Angular deviation: We measure how much removing the shared-pattern contribution changes the full gradient direction: θS=∠(g,gR)=arccos⁡(g⊤gR∥g∥2∥gR∥2).\theta_S = \angle(g,g_R) = \arccos\left( \frac{g^\top g_R}{\lVert g\rVert_2\lVert g_R\rVert_2} \right).
DAPODelTA
cos⁡(G+,S,G−,S)\cos(G_{+,S}, G_{-,S})0.990.97
cos⁡(gS,gR)\cos(g_S,g_R)-0.77-0.66
cos⁡(G+rand,G−rand)\cos(G_+^{\mathrm{rand}},G_-^{\mathrm{rand}})0.08-0.59
∥G+,S∥2/∥G−,S∥2\lVert G_{+,S}\rVert_2/ \lVert G_{-,S}\rVert_20.820.68
∥gS∥2/∥gR∥2\lVert g_S\rVert_2/\lVert g_R\rVert_20.800.59
∥gS∥2/∥g∥2\lVert g_S\rVert_2/\lVert g\rVert_21.260.78
∠(g,gR)\angle(g, g_R)53.3°35.8°
∠(g,gdelrand)\angle(g,g_{\mathrm{del}}^{\mathrm{rand}})40.7°20.2°

The positive- and negative-side shared gradients are highly aligned, whereas the count-matched random-token aggregates exhibit substantially lower cross-side alignment. Despite partial cancellation, the shared-pattern residual remains substantial relative to the remaining-token contribution and is negatively aligned with it. Removing shared-pattern contributions changes the gradient direction by 53.3° for DAPO and 35.8° for DelTA, compared with changes of 40.7° and 20.2° under count-matched random removal. DelTA exhibits a lower shared-to-remaining gradient norm ratio and a smaller angular deviation. These results provide direct evidence that independently identified shared-pattern tokens have a non-negligible net contribution and substantially affect the gradient direction.

AuthorsOpenReview ↗

Author Response (2/2)

Q2: Robustness to different rollouts


To evaluate the sensitivity of DelTA to rollout sampling, we conducted three independent training runs with different random seeds while keeping the training configuration fixed. For each run, we selected the checkpoint using AIME24 and evaluated it on the four benchmarks reported below.

DelTAAIME25AIME26HMMT25HMMT26Avg.
Run126.4628.1218.5420.2723.27
Run227.2928.7517.7120.0823.38
Run326.8827.9219.3821.0223.73

The results show that DelTA achieves highly consistent performance across runs, with a weighted average accuracy of 23.46 ± 0.24 (mean ± standard deviation). This suggests that the performance of DelTA is relatively stable under different rollout samples.

Q3: Comparison with related baselines


We have incorporated three additional related baselines: Entropy-Based Advantage Shaping [1], Probability-based Advantage Reweighting (AR) [2] and ResRL [3]. We also introduced two simpler gradient-based heuristic baselines to further examine whether DelTA's improvements can be achieved through simpler token-weighting strategies:

  • Grad Norm: Assigns higher weights to tokens with larger gradient-proxy norms.

  • Signed Gradient Alignment: Assigns higher weights to tokens whose advantage-signed gradient-proxy contributions are more aligned with the aggregate contribution of the other tokens.

For the two gradient-based baselines, we use the same gradient proxy, coefficient bounds, normalization scheme, and training budget as DelTA.

AIME25AIME26HMMT25HMMT26Avg.
DAPO23.3324.1712.0816.8619.05
Entropy-Based Advantage Shaping23.7523.5412.2916.4818.95
AR24.1723.7512.2916.2919.05
ResRL23.9623.9614.1716.8619.67
Grad Norm23.7523.1311.6716.1018.60
Signed Gradient Alignment24.1723.3312.0816.2918.90
DelTA26.4628.1218.5420.2723.27

DelTA outperforms all these baselines.

W2: Intervention-based evidence for DelTA


To obtain intervention-based evidence, we sampled 256 training questions for which both correct and incorrect rollouts were available, generating 16 responses per question. We computed DelTA’s token weights λ\lambda using all rollouts. For each question, we randomly selected one correct and one incorrect rollout. Within the first half of each selected response, we selected two non-overlapping 256-token spans with the highest and lowest mean λ\lambda.

For each selected span ss, we constructed two prefixes: pbeforep_{\mathrm{before}}, ending immediately before the span, and pafter=pbefore∥sp_{\mathrm{after}}=p_{\mathrm{before}}\Vert s, including the entire span. We independently sampled 16 continuations from each prefix using identical decoding and evaluation settings. We report the change in continuation accuracy. We compute the high-minus-low difference in continuation-accuracy changes within each question.

Source rolloutSpanpbeforep_{\mathrm{before}}pafterp_{\mathrm{after}}Δ\Delta
correcthigh λ62.5564.94+2.39
correctlow λ62.7461.43-1.32
incorrecthigh λ48.6345.31-3.32
incorrectlow λ48.8347.92-0.90

For correct rollouts, the paired high-minus-low difference is 3.71 percentage points, indicating that higher-weight spans yield more favorable continuation-accuracy changes even within successful responses.

For incorrect rollouts, the point estimates show a larger decrease in continuation accuracy for high-λ\lambda spans. The paired high-minus-low difference is −2.42 percentage points. This direction is consistent with the intended interpretation of negative-side weighting.

These results are consistent with the role of λ\lambda in modulating the signed advantage: high-λ\lambda spans show more favorable continuation-accuracy changes on the positive side and more unfavorable changes on the negative side. Together, these results provide additional intervention-based support for the credit-assignment interpretation.

References:

[1] Reasoning with exploration: An entropy perspective.

[2] Do not let low-probability tokens over-dominate in rl for llms.

[3] ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer FqSC,

Thank you again for your time and constructive feedback, which has been very helpful in improving our work.

As the discussion period nears its end (November 18 AoE), we wanted to leave a brief note to let you know we are following this thread closely. We hope our rebuttal has addressed your initial concerns. Should you have any remaining questions or need further clarification, we would be happy to discuss them further.

Thank you for your continued time and effort!

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.