- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DelTA, a token-level credit assignment method for RLVR. It interprets the local policy-gradient update as an implicit linear discriminator over token-gradient vectors, refines positive- and negative-side centroids, and uses distance-based token weights in a DAPO-style objective. An LM-head gradient proxy reduces computational cost. Experiments on seven math benchmarks report average gains of 3.26 and 2.62 points over the strongest baselines for Qwen3-8B and 14B, respectively. The paper also includes code-generation, out-of-domain, and alternative-backbone evaluations.
Critical strengths
-
The method shows consistent improvements across seven math benchmarks at both model scales, with additional positive results on code generation, OOD tasks, and Olmo3-7B. Appendix J.12 also shows that DelTA retains its advantage over SAPO under approximately matched wall-clock budgets, strengthening the empirical evidence.
-
The paper provides several meaningful ablations, including within-side-only weighting, random weights, and the main coefficient-design choices. Appendix J.8 evaluates the agreement between the LM-head gradient proxy and full-parameter gradients, while Appendix J.10 shows that token-level weighting performs better than trajectory-level weighting. These results provide useful evidence that the proposed token weights help optimization, although the underlying mechanism is not yet fully established.
Critical weaknesses
-
The proposed mechanism and its advantage over simpler weighting methods are not fully established. The paper argues that shared token-gradient patterns weaken discrimination between positive and negative tokens. However, the actual update depends on the mass-weighted centroid difference (Eq. 3), while Table 15 only measures centroid separation. Although Table 3 shows that refinement improves DelTA (23.27 vs. 19.97), it remains unclear whether simpler gradient-norm or gradient-alignment weighting could achieve similar results. A comparison under matched settings, together with a clearer distinction from ResRL [1], would better establish the proposed mechanism and its novelty.
-
Checkpoint selection and statistical evidence do not fully establish training robustness. Checkpoints are selected using AIME25, which is also part of the final evaluation (Appendix G). Appendix J.13 provides an alternative AIME24 selection check, but only for the reported 8B DAPO–DelTA comparison, not the strongest baselines. Moreover, Appendix I reports one training run per method, while the significance tests use 16 evaluation samples. These tests capture generation variability, not training-seed variability. Independent checkpoint selection and multiple training seeds would make the reported gains more convincing.
Questions for authors
-
Can the authors provide direct evidence that shared gradient patterns interfere with the actual policy update, beyond the centroid similarity analysis in Table 15?
-
Could the authors compare DelTA with simpler gradient-norm and signed gradient-alignment weighting under the same coefficient bounds, normalization, and training budget? How does the proposed mechanism differ from ResRL [1]?
-
Can the authors report the DelTA-versus-SAPO/FIPO comparisons using independently selected checkpoints or fixed training steps, together with training-seed variation for the main results?
[1] Lin, Zi-Han et al. “ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning.” ArXiv abs/2605.00380 (2026)
4 comments
Author Response (2/2)
Q1: Direct decomposition of shared-pattern contributions
We conducted a full-parameter gradient-decomposition experiment to directly examine how shared-pattern tokens contribute to the RLVR policy gradient. All objectives are evaluated at the same policy on the same sampled responses.
We randomly sampled 256 questions from the training set and generated 16 responses per question. We identified shared patterns independently of DelTA’s coefficients. We treated each contiguous 3-gram as a candidate pattern. For each question, we computed the occurrence rate of each 3-gram among positive- and negative-advantage responses, where the occurrence rate is defined as the fraction of responses containing that 3-gram. A 3-gram was identified as a shared pattern if it appeared in at least 30% of responses on both sides, with an absolute difference between the two occurrence rates no greater than 0.2. All token positions covered by at least one identified shared pattern were labeled as shared, while all other positions were labeled as remaining.
Let and denote the aggregated gradients of shared-pattern tokens on the positive- and negative-advantage sides. We aggregate token-level gradients using the absolute values of their corresponding objective coefficients. We retain the original coefficient magnitudes without normalizing the positive- and negative-side aggregates, thereby preserving their actual, potentially unequal gradient masses.
Their signed net contribution is
Let denote the net gradient contribution from all remaining tokens. The full gradient can then be decomposed as
We report the following metrics:
- Directional alignment: We report , measuring the directional alignment between the positive- and negative-side shared-pattern gradients, and , measuring the alignment between the net shared-pattern contribution and the remaining-token contribution.
- Matched random-token control: For each response, we uniformly sample the same number of token positions as identified shared-token positions. Let and denote the aggregated gradients of the sampled tokens on the positive- and negative-advantage sides. We report their cross-side cosine similarity, . We also compute the gradient remaining after removal of the sampled contributions, and report the random-deletion angle,
- Gradient norm ratios: We report to compare the magnitudes of the positive- and negative-side shared-pattern aggregates. We additionally report and , measuring the magnitude of the non-cancelled shared-pattern contribution relative to the remaining-token contribution and the full gradient.
- Angular deviation: We measure how much removing the shared-pattern contribution changes the full gradient direction:
| DAPO | DelTA | |
|---|---|---|
| 0.99 | 0.97 | |
| -0.77 | -0.66 | |
| 0.08 | -0.59 | |
| 0.82 | 0.68 | |
| 0.80 | 0.59 | |
| 1.26 | 0.78 | |
| 53.3° | 35.8° | |
| 40.7° | 20.2° |
The positive- and negative-side shared gradients are highly aligned, whereas the count-matched random-token aggregates exhibit substantially lower cross-side alignment. Despite partial cancellation, the shared-pattern residual remains substantial relative to the remaining-token contribution and is negatively aligned with it. Removing shared-pattern contributions changes the gradient direction by 53.3° for DAPO and 35.8° for DelTA, compared with changes of 40.7° and 20.2° under count-matched random removal. DelTA exhibits a lower shared-to-remaining gradient norm ratio and a smaller angular deviation. These results provide direct evidence that independently identified shared-pattern tokens have a non-negligible net contribution and substantially affect the gradient direction.
Official Review of Submission26076 by Reviewer G2eY
Thank you for the detailed rebuttal and the additional experiments. I appreciate the authors' efforts to address my concerns.
The additional baseline comparisons, gradient analysis, and independent training runs substantially address my concerns and strengthen the evidence for DelTA's effectiveness and robustness.
However, one concern remains regarding the interpretation of the shared-gradient analysis and its role in explaining DelTA's improvements. My original concern was whether shared gradient patterns actually weaken the induced discriminator, rather than simply making the positive- and negative-side centroids more similar.
The new decomposition shows that shared-pattern gradients make a substantial contribution to the net gradient and that DelTA reduces their relative influence in this analysis. This is useful evidence. However, a large contribution, even when negatively aligned with the remaining gradient, does not necessarily mean that it makes the policy update less effective. The remaining gradient is not necessarily a better update direction. More specifically, the decomposition does not directly show that shared-pattern gradients reduce the induced discriminator's ability to distinguish positive- from negative-advantage token gradients, or that reducing their influence explains DelTA's performance gains.
I acknowledge that the within-side-only ablation, refinement ablation, and token-selection experiments provide meaningful support for DelTA's weighting strategy. Together with the new decomposition, these results strengthen the proposed explanation, but do not fully establish the specific role of shared-gradient interference. I therefore consider this concern partially addressed and encourage the authors to make this distinction clear when discussing the proposed mechanism.
Overall, despite the remaining concern about the interpretation of the shared-gradient analysis, the rebuttal has substantially strengthened the paper, and I remain positive and maintain my original score.
We thank the reviewer for this important clarification. We agree that a substantial shared-gradient contribution, even when negatively aligned with the remaining gradient, does not necessarily imply that it harms policy optimization.
To investigate this question more directly, we conducted an additional training-time intervention that explicitly amplifies shared-pattern contributions during RL optimization.
Specifically, we identify shared-pattern tokens using the same 3-gram criterion described in our previous response. We then modify the DAPO objective by assigning 2× weight to shared-pattern tokens, while keeping the weights of all remaining tokens at 1×. The weighted loss is normalized by the sum of token weights to avoid trivially increasing the overall loss scale. All other training configurations remain unchanged.
Under the local policy-gradient formulation, this intervention increases the relative contribution of shared-pattern gradients, corresponding to a direction proportional to
compared with the original DAPO direction .
Due to time and computational resource constraints, we conducted this controlled intervention for 80 training steps and compared its performance with standard DAPO at the same training step, keeping all other training configurations identical.
The results are summarized below.
| Method | AIME25 | AIME26 | HMMT25 | HMMT26 | Avg. |
|---|---|---|---|---|---|
| DAPO | 21.88 | 22.71 | 10.63 | 15.15 | 17.53 |
| DAPO w/ Shared ×2 | 20.21 | 20.63 | 8.75 | 14.20 | 15.90 |
Amplifying shared-pattern contributions reduces the weighted average accuracy from 17.53 to 15.90, a decrease of 1.63 percentage points compared with standard DAPO.
Importantly, this experiment goes beyond characterizing gradient geometry: by directly intervening on shared-pattern contributions during RL training, we find that increasing their relative influence leads to lower downstream performance in this controlled training experiment. This provides intervention-based evidence that excessive emphasis on shared patterns can be detrimental to effective policy optimization, consistent with our shared-gradient interference hypothesis.
While this experiment does not directly measure the induced discriminator's classification performance, it provides evidence concerning the actual optimization consequences of emphasizing shared-pattern gradients, which is the central motivation of our method.
Together with the previous full-parameter gradient decomposition, refinement ablation, and token-selection experiments, these results provide complementary support for our proposed mechanism: shared patterns substantially influence the policy update, excessive emphasis on these patterns can impair training, and DelTA's selective weighting may help mitigate this interference.
We appreciate the reviewer's constructive feedback, which has helped us strengthen both the experimental evidence and the interpretation of our proposed mechanism.
Author Response (1/2)
We thank you for the constructive feedback. We provide clarifications and additional experimental results below in response to your comments. We will ensure that the experimental results and expanded discussions are incorporated into our revised manuscript. Unless otherwise specified, additional experiments use Qwen3-8B-Base and follow the four-benchmark analysis protocol in Section 5. Avg. denotes accuracy weighted by the number of questions in each benchmark.
W1&Q2: Comparison with simpler baselines and ResRL
We conducted additional experiments with ResRL [1] and these simpler credit-assignment baselines:
Grad Norm: Assigns higher weights to tokens with larger gradient-proxy norms.
Signed Gradient Alignment: Assigns higher weights to tokens whose advantage-signed gradient-proxy contributions are more aligned with the aggregate contribution of the other tokens.
To ensure a fair comparison, all methods use the same backbone, training data, rollout budget, optimizer, and number of training steps. For the two gradient-based baselines, we additionally use the same gradient proxy, coefficient bounds, and normalization scheme as DelTA, so that the comparison isolates the effect of the credit-assignment criterion.
DelTA consistently outperforms these baselines across all four benchmarks.
Furthermore, ResRL and DelTA differ in their weighting criteria and treatment of shared components. ResRL computes negative-advantage token weights from projection residuals relative to a subspace estimated from positive-advantage token representations, reducing penalties on negative-advantage tokens that overlap with positive-advantage responses. DelTA uses relative distances to the refined positive- and negative-side centroids of token-gradient proxies to assign weights on both advantage sides. Its objective is to emphasize outcome-discriminative gradient patterns in the policy update. Thus, ResRL protects shared components from excessive negative penalties, while DelTA emphasizes gradients that distinguish positive- from negative-advantage responses.
References:
[1] ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning.
W2-1&Q3-1: Checkpoint selection
To address the checkpoint selection concern, we additionally evaluate DelTA and its strongest baselines using checkpoints selected based on AIME24, which is independent of the evaluation benchmarks reported below.
For Qwen3-8B, we compare DelTA against SAPO using AIME24-selected checkpoints. For Qwen3-14B, we conduct the same comparison between DelTA and FIPO.
For DelTA, AIME24 selects the same checkpoint as the original selection criterion. With checkpoints selected using AIME24, DelTA outperforms SAPO on Qwen3-8B and FIPO on Qwen3-14B by 2.29 and 2.14 percentage points in weighted average accuracy. These results show that DelTA’s advantage persists under checkpoint selection independent of the reported evaluation benchmarks.
W2-2&Q3-2: Results across independent training runs
To assess the robustness of DelTA to training randomness, we conducted three independent training runs for both DelTA and SAPO using different random seeds while keeping all other training configurations fixed. For each run, we selected the checkpoint based on AIME24 performance and evaluated it on the four benchmarks reported below.
Across the three runs, DelTA achieves a weighted average accuracy of 23.46 ± 0.24 (mean ± standard deviation), consistently outperforming SAPO. Notably, DelTA outperforms SAPO in all three runs, demonstrating that its performance gains are reproducible across different random seeds rather than dependent on a particular training run.