acceptodds
Under review as a conference paper at ICLR 2027

Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed significant breakthroughs in complex LLM reasoning within verifiable domains, such as mathematics and programming. Recent efforts have sought to extend this paradigm to open-ended tasks by employing LLM-as-a-Judge to provide sequence-level rewards for policy optimization. However, such rewards are inherently sparse, failing to provide the fine-grained supervision required for generating complex, long-form trajectories. Furthermore, existing approaches treat the Judge as a black-box oracle, using only its verdict while discarding the rich intermediate feedback signals encoded in it. To address these limitations, we introduce Grad2Reward, a novel framework that extracts dense process rewards directly from the Judge's inference process via a single backward pass. Notably, Grad2Reward adopts a self-judging mechanism, allowing the policy to improve through its own evaluative signals without reliance on superior external Judges. To effectively leverage these dense rewards, we further introduce Token Advantage Policy Optimization (TAPO), which enables fine-grained policy updates. Extensive experiments demonstrate that Grad2Reward achieves superior performance and training efficiency across diverse open-ended tasks, affirming its effectiveness and broad applicability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
credit assignment, large language model, open-ended task, Judge, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bhT4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper tackles credit assignment in open-ended reasoning task, while standard LLM-as-a-Judge setups provide only sparse, sequence-level outcome rewards. The authors introduce GRAD2REWARD, a framework that extracts dense, token-level process rewards from an LLM Judge via gradient-based attribution (bt=gt⊤etb_t = g_t^\top e_t) using a single backward pass, requiring no trained Process Reward Model or external supervision. The framework couples this with a Self-Judging mechanism and Token Advantage Policy Optimization to perform fine-grained token-level policy updates.

Empirical results on two models and four benchmarks shows that GRAD2REWARD outperforms Vanilla-GRPO and RuscaRL and improves training efficiency at the same time. Ablation studies span attribution method, judge scale, multi-grader evaluation and other RL algorithms.

Critical strengths
  • Lightweight Token Credit Assignment via Self-Judging Gradients: The paper bypasses external judges and costly PRM annotation pipelines by freezing the base policy as a judge and deriving dense token rewards via a single backward pass (gt⊤etg_t^\top e_t), providing a computationally practical mechanism for open-ended self-improvement.
  • Rigorous Evaluation Protocol: The test grader is deliberately stronger than and disjoint from the training-time Judge, with a separate prompt template, and results are cross-checked across two additional graders, three independent runs, a second RL algorithm, out-of-distribution benchmarks
Critical weaknesses

1. The theoretical analysis does not support the method as implemented.

  • Section 4.3 derives an additive decomposition ∑tbt≈F(e)−F(0)\sum_t b_t \approx F(e) - F(0), then abandons the second-order term on empirical grounds. Eq. 4 applies softmax to bb, which destroys additivity, assigns positive reward shares to tokens with negative attribution, and makes the weight distribution depend on an arbitrary, model-specific scale.
  • The "local perturbation" justification for the zero baseline is also incorrect: zero-centered embedding distributions say nothing about ‖et‖‖e_t‖, and the Hessian ≈ −Fisher step is an expectation identity applied to a fixed verdict with a single-sample empirical Fisher. What remains after all approximations is Gradient × Input plus softmax, with no theoretical constraint on it.

2. Sensitivity to the Judge's verdict measures content quality, is asserted but never tested. Gradient attribution measures local sensitivity of a frozen Judge's log-prob to an input embedding. High influence can arise from rubric keywords, formatting, or positional preference rather than quality. The paper runs no faithfulness check, e.g., whether removing high-attribution tokens changes the verdict.

3. The gain is never attributed to gradient attribution. The method introduces two components, attribution and TAPO. TAPO's return Ri,t=∑s≥tri,sR_{i,t} = \sum_{s \ge t} r_{i,s} satisfies Ri,1=rR_{i,1} = r for every response, so early tokens receive essentially the sequence-level advantage and attribution only acts late in the sequence; with uniform α this already yields a front-loaded weighting r(T−t+1)/Tr(T−t+1)/T. No uniform or random attribution baseline under TAPO is reported, so the central claim has no supporting experiment.

4. The self-judging motivation lacks its control. Table 3 compares Judges of different strength, all with attribution. The claim "no need for a stronger external Judge" requires comparing self-judging + attribution against stronger Judge + sparse reward. That control is absent.

5. Results on Qwen3-4B do not support "consistent" improvement. On ResearchQA the method trails RuscaRL (75.3 vs 75.6); on RaR-Science all methods are within 1.5 points of the base model. Table 1 is single-run with no variance, while Table 4's standard deviations (0.3 to 1.3) make these differences uninterpretable.

6. The human expert validation is not usable as evidence. Table 6(a) reports token rewards of 2.21 and 1.37, which are impossible under Eq. 4 (α_t sums to 1, r ∈ [−1, 1]); the quantity measured is not the one defined. There is no baseline attribution method in 6(a), so the result cannot distinguish "aligns with experts" from "aligns with rubric keywords." Table 6(b) and Appendix F give no sample size, number of annotators, agreement, variance, or significance.

Questions for authors
  • Table 6(a) reports token rewards of 2.21 and 1.37. Under Eq. 4 these are impossible (Σα_t = 1, |r| ≤ 1). Could you explain what is actually being measured?
  • Are the results in Table 1 runnin on single or multiple seeds? If multiple, please give mean ± std; if single, then the "consistently" and "significantly" language on Qwen3-4B needs to be reconciled with Table 4's standard deviations.
  • Given R_{i,1} = r for every response, how much of the per-token advantage actually differs from the sequence-level one? Could you please provide the gap between two attribution method?
  • Reagrding the human study, could you provide further details, such as the number of annotators, samples, agreement, and whether span annotation was blind to rewards.
  • I suggest running TAPO with uniform attribution (α_t = 1/T) and with random attribution. Without this, nothing shows the gain comes from gradient attribution rather than from TAPO's return structure.
  • Vanilla-GRPO with Qwen2.5-7B and Qwen3-32B as training-time Judges could be added to Table 3. The self-judging claim is about replacing a stronger Judge, so the comparison must be against a stronger Judge with sparse reward, not a stronger Judge with attribution.
Reviewer_8mHZOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Grad2Reward redistributes rubric-level rewards over response tokens using gradient-times-input attribution from a frozen judge. Per-rubric attribution weights preserve the aggregate scalar reward, while token-level reward-to-go supplies the policy update. The main open-ended experiments use 1.5B and 4B students, independent evaluators, redistribution controls, repeated runs, and human checks. A separate mathematics experiment studies the approach with a trained outcome reward model.

Critical strengths
  1. A novel LLM RL algorithm of TAPO is proposed by using gradient-times-input attribution from a frozen judge for token-level credit assignment.

  2. The evaluation is not restricted to the training judge. Independent and alternative graders, repeated runs, and human checks offer several complementary checks on the gains. The human study is small, but it is still useful evidence beyond self-evaluation.

  3. The method obtains token-level feedback from existing rubric judgments without new process annotations. Preserving the scalar reward also makes the change conceptually clear: it changes credit assignment, rather than silently changing which responses receive a higher total score.

Critical weaknesses
  1. The token-level credit assignment is intuitive but relies on a strong assumption: tokens that influence the rubric judge’s verdict are also appropriate targets for reward or penalty. The theoretical approximation and existing attribution checks provide some support, but the reliability and stability of this mapping remain unclear. In particular, how sensitive are the token rewards to semantically equivalent rubric rewordings?

  2. LLM RL training is already compuiting-intensive, and Grad2Reward adds a backward attribution computation for each rubric criterion. With KK criteria, this requires KK judgment-specific gradient computations per response, beyond the policy update. This is a substantial practical concern for applying the method to large LLMs.

  3. The experiments use only Qwen2.5-1.5B and Qwen3-4B. The appendix includes a 7B mathematics experiment, but it uses a different reward-model setup. Results with larger students or MoE models under the same rubric-based setting would help establish the method’s broader applicability.

  4. The comparisons are largely limited to naive GRPO and RuscaRL, more competitive LLM RL baselines should be included.

  5. The evaluation covers four datasets concentrated in medical and scientific question answering. Broader evaluation would strengthen the generality claims, particularly on long-form tasks requiring evidence synthesis. Benchmarks such as ResearchRubrics [1] could make TAPO more convincing.

 

[1] Researchrubrics: A benchmark of prompts and rubrics for evaluating deep research agents

Questions for authors

Please see Critical Weaknesses.

Reviewer_bpymOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Grad2Reward, a framework that extracts PRM from the gradients of an LLM Judge through a single backward pass. Unlike existing methods that only use sequence-level rewards, Grad2Reward provides token-level feedback for more fine-grained policy optimization. Experiments on various open-ended tasks show that Grad2Reward improves both performance and training efficiency.

Critical strengths

The idea itself is interesting, e.g., using gradient attribute to provide the intermiedate feedback for the policy training. The author verify their proposed methods in several benchmarks.

Critical weaknesses

One weakness of this paper is that it misses some existing PRM baselines, such as PRIME, Math-Shepherd (MCTS-based rollout labeling), and rePIRL. It would be great if the authors could compare their method with these existing PRM methods in terms of both performance and training efficiency.

Besides, I have some concerns about why the gradient attribution-based method works. My understanding is that this method mainly redistributes the final outcome reward to the token level through gradient attribution, but it does not introduce any additional information, since the reward signal still comes from the final outcome. Why would this redistribution lead to better policy optimization? It would be great if the authors could provide more insights or theoretical analysis to explain why this approach works.

Questions for authors

See the weakness part

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.