- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper tackles credit assignment in open-ended reasoning task, while standard LLM-as-a-Judge setups provide only sparse, sequence-level outcome rewards. The authors introduce GRAD2REWARD, a framework that extracts dense, token-level process rewards from an LLM Judge via gradient-based attribution () using a single backward pass, requiring no trained Process Reward Model or external supervision. The framework couples this with a Self-Judging mechanism and Token Advantage Policy Optimization to perform fine-grained token-level policy updates.
Empirical results on two models and four benchmarks shows that GRAD2REWARD outperforms Vanilla-GRPO and RuscaRL and improves training efficiency at the same time. Ablation studies span attribution method, judge scale, multi-grader evaluation and other RL algorithms.
Critical strengths
- Lightweight Token Credit Assignment via Self-Judging Gradients: The paper bypasses external judges and costly PRM annotation pipelines by freezing the base policy as a judge and deriving dense token rewards via a single backward pass (), providing a computationally practical mechanism for open-ended self-improvement.
- Rigorous Evaluation Protocol: The test grader is deliberately stronger than and disjoint from the training-time Judge, with a separate prompt template, and results are cross-checked across two additional graders, three independent runs, a second RL algorithm, out-of-distribution benchmarks
Critical weaknesses
1. The theoretical analysis does not support the method as implemented.
- Section 4.3 derives an additive decomposition , then abandons the second-order term on empirical grounds. Eq. 4 applies softmax to , which destroys additivity, assigns positive reward shares to tokens with negative attribution, and makes the weight distribution depend on an arbitrary, model-specific scale.
- The "local perturbation" justification for the zero baseline is also incorrect: zero-centered embedding distributions say nothing about , and the Hessian ≈ −Fisher step is an expectation identity applied to a fixed verdict with a single-sample empirical Fisher. What remains after all approximations is Gradient × Input plus softmax, with no theoretical constraint on it.
2. Sensitivity to the Judge's verdict measures content quality, is asserted but never tested. Gradient attribution measures local sensitivity of a frozen Judge's log-prob to an input embedding. High influence can arise from rubric keywords, formatting, or positional preference rather than quality. The paper runs no faithfulness check, e.g., whether removing high-attribution tokens changes the verdict.
3. The gain is never attributed to gradient attribution. The method introduces two components, attribution and TAPO. TAPO's return satisfies for every response, so early tokens receive essentially the sequence-level advantage and attribution only acts late in the sequence; with uniform α this already yields a front-loaded weighting . No uniform or random attribution baseline under TAPO is reported, so the central claim has no supporting experiment.
4. The self-judging motivation lacks its control. Table 3 compares Judges of different strength, all with attribution. The claim "no need for a stronger external Judge" requires comparing self-judging + attribution against stronger Judge + sparse reward. That control is absent.
5. Results on Qwen3-4B do not support "consistent" improvement. On ResearchQA the method trails RuscaRL (75.3 vs 75.6); on RaR-Science all methods are within 1.5 points of the base model. Table 1 is single-run with no variance, while Table 4's standard deviations (0.3 to 1.3) make these differences uninterpretable.
6. The human expert validation is not usable as evidence. Table 6(a) reports token rewards of 2.21 and 1.37, which are impossible under Eq. 4 (α_t sums to 1, r ∈ [−1, 1]); the quantity measured is not the one defined. There is no baseline attribution method in 6(a), so the result cannot distinguish "aligns with experts" from "aligns with rubric keywords." Table 6(b) and Appendix F give no sample size, number of annotators, agreement, variance, or significance.
Questions for authors
- Table 6(a) reports token rewards of 2.21 and 1.37. Under Eq. 4 these are impossible (Σα_t = 1, |r| ≤ 1). Could you explain what is actually being measured?
- Are the results in Table 1 runnin on single or multiple seeds? If multiple, please give mean ± std; if single, then the "consistently" and "significantly" language on Qwen3-4B needs to be reconciled with Table 4's standard deviations.
- Given R_{i,1} = r for every response, how much of the per-token advantage actually differs from the sequence-level one? Could you please provide the gap between two attribution method?
- Reagrding the human study, could you provide further details, such as the number of annotators, samples, agreement, and whether span annotation was blind to rewards.
- I suggest running TAPO with uniform attribution (α_t = 1/T) and with random attribution. Without this, nothing shows the gain comes from gradient attribution rather than from TAPO's return structure.
- Vanilla-GRPO with Qwen2.5-7B and Qwen3-32B as training-time Judges could be added to Table 3. The self-judging claim is about replacing a stronger Judge, so the comparison must be against a stronger Judge with sparse reward, not a stronger Judge with attribution.