- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
GRPO assigns the same response advantage to every token despite differences in their influence on the final answer. GRAIL computes gradient activation saliency from the prediction loss of the generated answer and uses it to reweight token advantages. Scores are log transformed, standardized, and clipped, with separate rules for answer tokens, delimiters, and tokens after the answer. The strongest reported setting applies weighting only to incorrect rollouts. Across five models and six mathematical benchmarks, GRAIL improves average accuracy by approximately 3.60 percentage points and Pass@3 by 3.05 points over GRPO.
Critical strengths
GRAIL provides token weighting without an external teacher or process reward model. Gains across five models support its empirical usefulness. Comparisons with entropy weighting, OAR-G, and answer weighting alone help assess the full method, while the reported training times make its additional cost explicit.
Critical weaknesses
-
The paper does not show the original RL gradients across tokens or how GRAIL changes their distribution. A shared advantage does not imply equal token gradients. It therefore remains unclear whether the saliency weights provide additional information or mainly amplify existing gradient differences. Moreover, sensitivity to a generated answer does not establish that a token is erroneous or deserves stronger punishment.
-
Although Table 2 includes entropy weighting and OAR-G, the comparison omits relevant methods that explicitly analyze and reshape token gradients, particularly AR/Lopti [c1] and DelTA [c2]. These comparisons would help establish the benefit of answer loss saliency over existing gradient based approaches.
-
The additional computation and memory requirements are not analyzed as response length increases. The reported timing comparison is useful but does not establish practicality for longer reasoning traces.
-
The method depends on identifying an explicit final answer span, but the handling of missing or malformed answer spans is not specified. Without this span, the answer loss and structural weight corrections are undefined as described, leaving an important part of the training procedure unclear.
[c1] Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs.
[c2] DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards.
Questions for authors
-
How are the original RL gradient magnitudes distributed across tokens, and how does GRAIL change this distribution? How strongly do saliency scores correlate with the original gradient magnitudes?
-
Do high saliency tokens coincide with actual reasoning errors rather than correct premises or other information used to predict an incorrect answer?
-
How does GRAIL compare with AR/Lopti [c1] and DelTA [c2] under matched training conditions?
-
How do the additional time complexity and memory requirements scale with response length? What overhead is observed at different lengths?
-
How are responses without a valid final answer span handled? How frequently does this occur, and are these responses excluded, assigned uniform weights, or processed using another rule?
Dear Reviewers,
Thank you for the review. The authors have responded to the points you raised, and the response period closes on 18 November (AoE). I would be grateful if you could post a brief reply on whether your concerns are resolved, partly resolved, or still open, so that the authors have a fair chance to address them. If anything still needs clarifying, please ask now while they can still respond.
Regards,
AC