acceptodds
Under review as a conference paper at ICLR 2027

GRAIL: Gradient-Reweighted Advantages for Reinforcement Learning with Verifiable Rewards

Abstract

Reinforcement learning with verifiable rewards (e.g. GRPO) is now a common way to improve mathematical reasoning in Large Language Models (LLMs). However, current methods usually broadcast one sequence-level advantage to all tokens, or use costly process reward models (PRMs) for step-level supervision. Uniform advantage distribution assumes that all tokens contribute equally to the final reward. This dilutes the gradient signal, since flawed reasoning steps and filler words are updated as strongly as valid logical inferences. To address this, we introduce Gradient-Reweighted Advantage (GRAIL), an intrinsic token-wise advantage reweighting method. GRAIL uses gradient-activation saliency to place more weight on tokens that are more locally sensitive to the final answer. Evaluations across five models from the Qwen3, R1-distilled and OctoThinker families show that GRAIL consistently outperforms GRPO. GRAIL achieved an average improvement of 3.60% in accuracy and 3.05% in Pass@3, demonstrating that fine-grained reasoning alignment can be achieved without process-level supervision. Our code is available at https://anonymous.4open.science/r/GRAIL-C687

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Math Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_F2RjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GRPO assigns the same response advantage to every token despite differences in their influence on the final answer. GRAIL computes gradient activation saliency from the prediction loss of the generated answer and uses it to reweight token advantages. Scores are log transformed, standardized, and clipped, with separate rules for answer tokens, delimiters, and tokens after the answer. The strongest reported setting applies weighting only to incorrect rollouts. Across five models and six mathematical benchmarks, GRAIL improves average accuracy by approximately 3.60 percentage points and Pass@3 by 3.05 points over GRPO.

Critical strengths

GRAIL provides token weighting without an external teacher or process reward model. Gains across five models support its empirical usefulness. Comparisons with entropy weighting, OAR-G, and answer weighting alone help assess the full method, while the reported training times make its additional cost explicit.

Critical weaknesses
  1. The paper does not show the original RL gradients across tokens or how GRAIL changes their distribution. A shared advantage does not imply equal token gradients. It therefore remains unclear whether the saliency weights provide additional information or mainly amplify existing gradient differences. Moreover, sensitivity to a generated answer does not establish that a token is erroneous or deserves stronger punishment.

  2. Although Table 2 includes entropy weighting and OAR-G, the comparison omits relevant methods that explicitly analyze and reshape token gradients, particularly AR/Lopti [c1] and DelTA [c2]. These comparisons would help establish the benefit of answer loss saliency over existing gradient based approaches.

  3. The additional computation and memory requirements are not analyzed as response length increases. The reported timing comparison is useful but does not establish practicality for longer reasoning traces.

  4. The method depends on identifying an explicit final answer span, but the handling of missing or malformed answer spans is not specified. Without this span, the answer loss and structural weight corrections are undefined as described, leaving an important part of the training procedure unclear.

[c1] Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs.

[c2] DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards.

Questions for authors
  1. How are the original RL gradient magnitudes distributed across tokens, and how does GRAIL change this distribution? How strongly do saliency scores correlate with the original gradient magnitudes?

  2. Do high saliency tokens coincide with actual reasoning errors rather than correct premises or other information used to predict an incorrect answer?

  3. How does GRAIL compare with AR/Lopti [c1] and DelTA [c2] under matched training conditions?

  4. How do the additional time complexity and memory requirements scale with response length? What overhead is observed at different lengths?

  5. How are responses without a valid final answer span handled? How frequently does this occur, and are these responses excluded, assigned uniform weights, or processed using another rule?

Reviewer_fcxLOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses coarse credit assignment in GRPO, where one sequence-level advantage is broadcast to every token. The paper proposes GRAIL, which reweights each token’s advantage by a gradient * activation saliency score. The gradient of the final-answer log-likelihood with respect to the token’s input embedding is multiplied element-wise by the embedding, log-transformed, standardized, and clipped to [0.5, 5.0]. Answer tokens are fixed at the maximum weight, and delimiter and post-answer tokens at the neutral weight. The weights are stop-gradient.

On DeepMath-103K with Dr. GRPO, GRAIL is evaluated on five models from three families across six math benchmarks. It outperforms GRPO by 3.60 points in average accuracy and 3.05 points in Pass@3. On Qwen3-4B/8B, it also outperforms an entropy-based reweighting baseline and OAR-G. Ablations cover the placement of the differentiable leaf and symmetric versus asymmetric weighting. The best results come from applying the saliency weights only to incorrect rollouts.

Critical strengths
  • S1: GRAIL improves over GRPO on all five models and on most of the six benchmarks, in both average accuracy and Pass@3.

  • S2: The evaluation covers three model families and includes a leaf-placement ablation, a positional analysis, and matched wall-clock comparisons against the closest gradient-based baseline. The coverage is good and on par with prior art.

Critical weaknesses

W1. Limited novelty and an outdated framing.

The motivation (lines 43–47) states that current GRPO-style methods broadcast one sequence-level advantage to all tokens and treat all tokens equally. Token-level advantage shaping is already an established line of work [1,2,3,4,5]. The paper’s own Table 2 compares to two of these, so the framing is inconsistent with its own experiments. What remains is a specific saliency estimator, gradient × activation at the input embeddings, combined with structural weight corrections. The paper's differences from OAR-G are a different sensitivity estimator and lower cost. This is incremental, and GRAIL's cost is still 38% over GRPO. Note that accessing the internal embeddings and gradients severely slows down the RLVR training process

W2. Several problems exist with the current experimental design, baselines, ablation and potential unfair comparison The reported variant is chosen post hoc, and its behavior is unexplained. Table 1 reports the best setting from Section 5.3.2: weights applied only to wrong rollouts. This setting was selected on the test benchmarks, and the symmetric method defined in Section 3 is 0.7–3.3 points worse (Table 4). The asymmetry is also counterintuitive and unjustified: if saliency identifies tokens responsible for the outcome, weighting correct rollouts should help too, yet "Correct Rollouts Only" is the worst setting by a wide margin. The positional analysis in Section 5.4 only describes the weight distributions, it does not explain why upweighting salient tokens in correct rollouts hurts. Table 4 also omits GRPO. Comparing with Table 1, "Correct Rollouts Only" falls below GRPO on Qwen3-4B (53.33 vs. 55.20) and roughly matches it on Qwen3-8B. The method can therefore hurt performance depending on where it is applied, and the paper does not explain why. The gains may come from the weight magnitude, not from saliency. Weights span [0.5, 5.0], answer tokens are fixed at 5.0, and wrong rollouts receive higher mean weights (Figure 3). GRAIL therefore increases the loss magnitude relative to GRPO, especially on negative-advantage rollouts. The two simple controls that would rule this out are missing: GRPO with its loss uniformly rescaled to match GRAIL's total weight, and GRPO with GRAIL’s per-sequence weight sum spread uniformly or randomly across tokens. If either of these matches GRAIL, the gradient computation through the embeddings is unnecessary and the paper's central claim does not hold. The baselines are underspecified and the tables are inconsistent. No training details are given for the entropy baseline, and "Answer-Weighting-Only" is an ablation rather than a baseline. The "All Rollouts" row in Table 4 matches the "Input Embeddings" row in Table 3, while "Wrong Rollouts Only" matches Table 1, so the two ablations appear to use different base configurations without explanation. Several notable token-level methods are also absent [1,2,4] Narrow scope. Only mathematical reasoning is evaluated. Code generation, scientific reasoning, and other verifiable tasks are not considered, so the generality claimed in the conclusion is untested. Missing training dynamics. As stated above, since the method dives into extracting internal representations and gradients, it severely affects the RLVR training process efficiency, especially for frameworks that utilized flash-attn. Hence, efficiency discussion, training dynamics are necessary to get a better view of the training progress. Also I would like to see the saliency scores distribution across the tokens, to see what type of tokens gets upweighted and what gets downweighted, which is an important analysis IMO.

References:

[1] Cheng, Daixuan, Shaohan Huang, Xuekai Zhu, Bo Dai, Xin Zhao, Zhenliang Zhang, and Furu Wei. "Reasoning with exploration: An entropy perspective." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 36, pp. 30377-30385. 2026.

[2] Hongze Tan, Jianfei Pan, Jinghao Lin, Tao Chen, Zhihang Zheng, Zhihao Tang, Haihua Yang. "GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy." arXiv preprint arXiv:2508.04349 (2025)

[3] Wang, Shenzhi, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang et al. "Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning." Advances in Neural Information Processing Systems 38 (2026): 115452-115486.

[4] Le, Thanh-Long V., Myeongho Jeon, Kim Vu, Viet Lai, and Eunho Yang. "No prompt left behind: Exploiting zero-variance prompts in llm reinforcement learning via entropy-guided advantage shaping." In International Conference on Learning Representations, vol. 2026, pp. 121956-121982. 2026.

[5] Ziheng Li, Liu Kang, Feng Xiao, Luxi Xing, Qingyi Si, Zhuoran Li, Weikang Gong, Deqing Yang, Yanghua Xiao, and Hongcheng Guo. Outcome-grounded advantage reshaping for fine-grained credit assignment in mathematical reasoning. arXiv preprint arXiv:2601.07408, 2026.

Questions for authors

N/A

Reviewer_59ibOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GRAIL, a token-level advantage reweighting method for reinforcement learning with verifiable rewards. It computes gradient-times-input saliency from the negative log-likelihood of the model's generated final answer, then uses normalized and clipped saliency scores to reweight the sequence-level advantage. Experiments on five models and six mathematical reasoning benchmarks report improvement over GRPO. Applying the weights only to incorrect rollouts gives the strongest results on the two models used for this ablation.

Critical strengths
  1. This paper provides consistent evidence that its reweighting method improves mathematical reasoning. The average accuracy gains across five models from three families make the empirical results more convincing than an improvement on a single backbone.

  2. This paper includes useful comparisons and ablations. Comparisons with OAR-G, entropy-based reweighting, and the Answer-Weighting-Only control help establish the practical value of the proposed signal.

Critical weaknesses
  1. The substantive advance over existing token-level advantage methods is not fully established. OAR-G [1] already uses final-answer sensitivity and gradient-based attribution to reweight sequence advantages. GRAIL simplifies the attribution objective and reports better accuracy with lower overhead, which is useful. However, KTAE [2] already estimates token-level advantage corrections from group statistics, and NTHR [3] selectively reduces penalties on tokens in incorrect responses. Neither is discussed or evaluated, despite their direct relevance, particularly to the finding that reweighting incorrect rollouts works best.

  2. The evidence does not establish that the saliency scores identify reasoning errors. The attribution loss measures the likelihood of the model's own generated answer, including when that answer is incorrect. A token can strongly influence this likelihood without being the source of an error.


[1] Li et al., Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning. https://arxiv.org/abs/2601.07408

[2] Sun et al., KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoning. https://arxiv.org/abs/2505.16826

[3] Deng et al., On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization. https://arxiv.org/abs/2505.18830

Questions for authors
  1. How does GRAIL compare with KTAE and NTHR under matched training settings? What evidence isolates the benefit of the answer-loss attribution signal from the other differences between GRAIL and OAR-G?

  2. Do high-saliency tokens correspond to actual reasoning errors?

  3. The Answer-Weighting-Only control is useful, but it does not isolate the value of the attribution scores from other ways of redistributing negative updates over reasoning tokens. Shuffled-weight or position-matched controls would help determine whether the gains depend on meaningful token attribution. Does GRAIL outperform shuffled-weight or position-matched controls on incorrect rollouts?

Comments on the paper

1 comment

Dear Reviewers,

Thank you for the review. The authors have responded to the points you raised, and the response period closes on 18 November (AoE). I would be grateful if you could post a brief reply on whether your concerns are resolved, partly resolved, or still open, so that the authors have a fair chance to address them. If anything still needs clarifying, please ask now while they can still respond.

Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.