- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies credit assignment when one LLM response contains multiple candidates and training rewards depend on the utility of the candidate set. ShapE-GRPO treats candidates as cooperative-game players, multiplies each candidate's Shapley value by the candidate count K, and broadcasts the result to its tokens. Reasoning tokens retain the sequence reward, while advantage normalization uses the original GRPO group statistics. For nonnegative max-based set rewards, the paper derives an exact allocation formula and polynomial computation cost. Experiments on ACLSum, DS-1000, and two Netflix recommendation settings, using Qwen3-8B and Llama 3.1-8B-Instruct, compare against GRPO, WTA, and DCR and report improved performance and convergence in training steps.
Critical strengths
-
A practical and computationally simple candidate-level allocation. Section 3 and Theorem 4.1 turn set-level credit assignment into a token-advantage modification that can be integrated into GRPO. For nonnegative max games, the sorted decomposition and the K/m rule for binary rewards are clear and directly computable. This is a useful method for multi-candidate generation tasks where candidate scores are available and is the main methodological reason to consider acceptance.
-
Positive evidence across tasks and model families. In Table 1, Qwen3-8B improves from GRPO's 3.710 to 4.141 on Netflix with history; in Table 4, Llama improves from 0.205 to 0.278 on DS-1000. Comparisons with DCR and WTA, plus the increased-rollout experiment in Appendix C.4, strengthen the empirical motivation. The recommendation gains are particularly promising.
Critical weaknesses
-
Conservation of summed advantages does not establish unbiased policy gradients, and the theoretical scope is overstated. Proposition 4.1 / Appendix B.1 proves only conservation of ∑ₜ Aₜ for equal-length candidates. Policy updates additionally involve token-dependent ∇θ log πθ, importance ratios, and clipping, so this identity does not imply the claim on page 5 that no additional bias is introduced relative to GRPO. The counterexample below uses equal-length candidates and binary max rewards, yet yields a nonzero update when set reward is constant. With unequal candidate lengths, even the proved conservation identity does not generally hold; Figure 2 only characterizes length variation for summarization. Furthermore, Proposition 4.2 is stated for a general nonnegative reward function, but Appendix B.2 relies on the max structure: zero standalone reward does not imply a null player in a general cooperative game. Polynomial exact computation likewise relies on the special max structure.
-
The experiments do not isolate the allocation rule's effect. Appendix C.3 states that there is one rollout per prompt. If this means G=1, Appendix B.2 sets mean(R)=0; together with the stated std=1 handling for zero variance, the GRPO baseline becomes a sequence-reward update without group-relative centering. Table 2 does not explicitly specify G, and the fourfold-rollout experiment in C.4 lacks complete settings and repetitions. More importantly, C.3 penalizes only excess thinking tokens for ShapE but subtracts the penalty from the entire sequence reward for GRPO, changing reward treatment alongside candidate credit assignment. WTA's description also leaves its K scaling unclear: for binary rewards, a scale-matched WTA that splits ties should produce exactly the same candidate rewards as ShapE, requiring an explanation of the DS-1000 difference. Finally, no multi-seed results or confidence intervals are reported. Qwen's DS-1000 gain over DCR is only 0.005, equivalent to two problems if each of the 400 test problems is evaluated once. The FF test on signal distributions does not establish significance of performance differences.
Questions for authors
Q1 — Theory: What random variable and objective does “unbiased” refer to, and how does Proposition 4.1 establish a policy-gradient property? Please address the counterexample below and state the reward classes under which Proposition 4.2 and the complexity claims actually hold.
Q2 — Training comparability: What is the group size G per prompt in the main experiments, and how are mean/std computed for G=1 and equal-reward groups? Please provide matched experiments with effective group-relative normalization and identical length-penalty treatment, together with full settings for the fourfold-rollout experiment.
Q3 — Baseline mechanism: Does WTA include the K multiplier, and how are ties, reasoning rewards, group statistics, and length penalties handled? With binary DS-1000 rewards, scale-matched WTA and ShapE should coincide; if results still differ, what exact implementation difference explains this?
Q4 — Strength of evidence: Can you report multiple training seeds, test uncertainty, and actual training time and candidate-scoring cost to reach the same target score? These would materially affect the claims of reliable gains and faster convergence, especially for small improvements over DCR.