acceptodds
Under review as a conference paper at ICLR 2027

ShapE-GRPO: Shapley-Enhanced Reward Allocation for Multi-Candidate LLM Training

Abstract

In user-agent interaction scenarios such as recommendation, brainstorming, and code suggestion, Large Language Models (LLMs) often generate sets of candidate recommendations where the objective is to maximize the collective utility of the entire set rather than individual candidates independently. However, existing reinforcement learning post-training paradigms, such as Group Relative Policy Optimization (GRPO), typically assign the same set-level scalar reward to every candidate in the set. This leads to noisy training signals where poor candidates free-ride on the high reward produced by a single strong peer, resulting in suboptimal exploration. To address this, we propose Shapley-Enhanced GRPO (ShapE-GRPO). By leveraging the permutation-invariant nature of set-level utility, we derive a Shapley-enhanced formulation from cooperative game theory to decompose set-level rewards into granular, candidate-specific signals. We show that our formulation preserves the fundamental axioms of the Shapley value while remaining computationally efficient with polynomial-time complexity. Empirically, ShapE-GRPO consistently outperforms standard GRPO across diverse datasets with accelerated convergence during training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
GRPO, Shapley Value, Reward Allocation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3PnUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies credit assignment when one LLM response contains multiple candidates and training rewards depend on the utility of the candidate set. ShapE-GRPO treats candidates as cooperative-game players, multiplies each candidate's Shapley value by the candidate count K, and broadcasts the result to its tokens. Reasoning tokens retain the sequence reward, while advantage normalization uses the original GRPO group statistics. For nonnegative max-based set rewards, the paper derives an exact allocation formula and polynomial computation cost. Experiments on ACLSum, DS-1000, and two Netflix recommendation settings, using Qwen3-8B and Llama 3.1-8B-Instruct, compare against GRPO, WTA, and DCR and report improved performance and convergence in training steps.

Critical strengths
  1. A practical and computationally simple candidate-level allocation. Section 3 and Theorem 4.1 turn set-level credit assignment into a token-advantage modification that can be integrated into GRPO. For nonnegative max games, the sorted decomposition and the K/m rule for binary rewards are clear and directly computable. This is a useful method for multi-candidate generation tasks where candidate scores are available and is the main methodological reason to consider acceptance.

  2. Positive evidence across tasks and model families. In Table 1, Qwen3-8B improves from GRPO's 3.710 to 4.141 on Netflix with history; in Table 4, Llama improves from 0.205 to 0.278 on DS-1000. Comparisons with DCR and WTA, plus the increased-rollout experiment in Appendix C.4, strengthen the empirical motivation. The recommendation gains are particularly promising.

Critical weaknesses
  1. Conservation of summed advantages does not establish unbiased policy gradients, and the theoretical scope is overstated. Proposition 4.1 / Appendix B.1 proves only conservation of ∑ₜ Aₜ for equal-length candidates. Policy updates additionally involve token-dependent ∇θ log πθ, importance ratios, and clipping, so this identity does not imply the claim on page 5 that no additional bias is introduced relative to GRPO. The counterexample below uses equal-length candidates and binary max rewards, yet yields a nonzero update when set reward is constant. With unequal candidate lengths, even the proved conservation identity does not generally hold; Figure 2 only characterizes length variation for summarization. Furthermore, Proposition 4.2 is stated for a general nonnegative reward function, but Appendix B.2 relies on the max structure: zero standalone reward does not imply a null player in a general cooperative game. Polynomial exact computation likewise relies on the special max structure.

  2. The experiments do not isolate the allocation rule's effect. Appendix C.3 states that there is one rollout per prompt. If this means G=1, Appendix B.2 sets mean(R)=0; together with the stated std=1 handling for zero variance, the GRPO baseline becomes a sequence-reward update without group-relative centering. Table 2 does not explicitly specify G, and the fourfold-rollout experiment in C.4 lacks complete settings and repetitions. More importantly, C.3 penalizes only excess thinking tokens for ShapE but subtracts the penalty from the entire sequence reward for GRPO, changing reward treatment alongside candidate credit assignment. WTA's description also leaves its K scaling unclear: for binary rewards, a scale-matched WTA that splits ties should produce exactly the same candidate rewards as ShapE, requiring an explanation of the DS-1000 difference. Finally, no multi-seed results or confidence intervals are reported. Qwen's DS-1000 gain over DCR is only 0.005, equivalent to two problems if each of the 400 test problems is evaluated once. The FF test on signal distributions does not establish significance of performance differences.

Questions for authors

Q1 — Theory: What random variable and objective does “unbiased” refer to, and how does Proposition 4.1 establish a policy-gradient property? Please address the counterexample below and state the reward classes under which Proposition 4.2 and the complexity claims actually hold.

Q2 — Training comparability: What is the group size G per prompt in the main experiments, and how are mean/std computed for G=1 and equal-reward groups? Please provide matched experiments with effective group-relative normalization and identical length-penalty treatment, together with full settings for the fourfold-rollout experiment.

Q3 — Baseline mechanism: Does WTA include the K multiplier, and how are ties, reasoning rewards, group statistics, and length penalties handled? With binary DS-1000 rewards, scale-matched WTA and ShapE should coincide; if results still differ, what exact implementation difference explains this?

Q4 — Strength of evidence: Can you report multiple training seeds, test uncertainty, and actual training time and candidate-scoring cost to reach the same target score? These would materially affect the claims of reliable gains and faster convergence, especially for small improvements over DCR.

Reviewer_AFJPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ShapE-GRPO to address the issue where low-quality candidates free ride under a unified set reward when large language models generate multiple candidate answers with GRPO style training method. Motivated by the Shapley value from cooperative game, ShapE-GRPO allocates rewards according to the Shapley values of candidates. Under the setting where the set reward takes the highest score among candidates, the authors derive an efficient exact calculation formula. Experiments on several datasets demonstrate performance gains and faster convergence compared with standard GRPO.

Critical strengths

S1: The research problem is practically meaningful.

Candidate-level reward allocation fits well with the task structure of multi-candidate generation. The method has a clear cooperative game interpretation and can be computed efficiently under the studied max-reward setting, demonstrating good practicality.

Critical weaknesses

W1: The experiments have not sufficiently isolated the contribution of the Shapley allocation itself.

Under binary rewards, if Winner-Takes-All (WTA) splits rewards equally among tied correct candidates and applies the same scaling, its candidate rewards should align with the method proposed in this paper.

Besides, Appendix C.3 applies length penalties of different granularities to GRPO and ShapE-GRPO, which may also affect the comparison results. Controlled experiments with these settings unified would be more convincing.

W2: The theoretical analysis is insufficient to support conclusions.

Proposition 4.1 only proves that the sum of advantage values remains unchanged under equal-length candidates, yet the policy gradient also depends on the log-probability gradients of each token. Hence, it cannot be concluded that the update will not introduce additional bias.

Furthermore, the proofs for efficient computation and the absence of positive incentives for zero-score candidates rely on the max-reward structure, and the scope of applicability for general set rewards needs to be more clearly defined.

Questions for authors

Q1: Although lots of fine-grained reward modeling methods are introduced in Appendix A.1, only GRPO and WTA are compared in the main experiments. What would the model perform under other stronger baselines?

Q2: Appendix C.3 states one rollout per prompt yields four candidates. Does this correspond to GRPO group size G=1 (one full response per prompt)? If so, how is the within-group reward mean and std calculated? Does our method still hold consistent benefits with multiple sampled responses per prompt?

Reviewer_ycRCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes candidate-level reward allocation for multi-candidate LLM generation. Scaled Shapley values redistribute max-based set rewards among candidates, while the reasoning segment retains the sequence reward. It derives an efficient exact computation and evaluates the method on summarization, code generation, and movie recommendation.

Critical strengths

The allocation addresses a concrete credit-assignment problem in set-valued generation and can be computed efficiently for max-based utility. Results across multiple tasks and comparisons with candidate-level baselines provide encouraging evidence of practical usefulness.

Critical weaknesses
  1. Proposition 4.1 conserves the sum of token advantages, but this does not establish an unbiased policy gradient: the token-level score functions differ. The surrounding claim of no additional bias therefore requires a separate justification or a narrower statement (pp. 5–6).
  2. The attribution of gains to Shapley allocation needs stronger controls. With binary rewards, tie-split WTA and ShapE share the same candidate credit pattern up to a factor of K. Appendix C.3 also applies length penalties differently to ShapE and GRPO. These differences leave reward scale and penalty placement as alternative explanations (pp. 8, 23).
Questions for authors
  1. Can you justify the policy-gradient bias claim beyond conservation of the sum of advantages, or restrict the claim to what Proposition 4.1 actually establishes?
  2. Do the gains persist with scale-matched WTA rewards and identical length penalties across methods? Please also clarify the mean/std convention used with the single rollout per prompt in Appendix C.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.