- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies step-level credit assignment for multi-step search agents. It proposes Sequential Shapley, which extends standard Shapley attribution by incorporating both interaction effects among search steps and sequential dependencies, so that early steps can receive credit for enabling useful later searches. The resulting step rewards are integrated with GRPO to form SPGRPO. Experiments on seven search-augmented QA benchmarks with Qwen2.5-3B/7B show consistent improvements over outcome-level RL and existing step-level methods。
Critical strengths
This paper proposes a reasonable design of sequential Shapley for effective credit assignment in search agent.
Theoretical analysis of sequential Shapley is provided.
Experiments are provided to justify the proposed method.
Critical weaknesses
This paper is a bit overclaim on the novelty of sequential shapley. The concept of sequential shapley was not introduced for the first time by this paper. Instead, SSVPO (Li et al., 2026) has already formulated sequential Shapley-based credit assignment for LLM reasoning. While this paper differs in how sequential structure is modeled, the introduction repeatedly describes sequential Shapley itself as a novel framework. Therefore, the prior work SSVPO should be discussed in the introduction instead of just cited in the experimental part as a compared method.
In Table 1, SSVPO should be included in comparison, especially comparisons on the 7B model.
The experimental comparisons are limitted only to Qwen2.5 models. Comparisons on newer Qwen models and other model families should be included.
Questions for authors
Could the authors empirically quantify M
max S ∣ R ( S ) ∣ M=max S ∣R(S)∣, on actual search trajectories? How does the value of M M compared to ϕ s e q ϕ seq and ϕ s t a n d a r d ϕ standard empirically?
How does the proposed method perform on newer Qwen models and other model families?
4 comments
Question 2. How does the proposed method perform on newer Qwen models and other model families?
Response. SPGRPO achieves the highest average performance among the compared methods on both Qwen3-4B (48.1) and Llama-3.2-3B (47.8). These scores improve over standard Shapley by 0.8 and 1.6 points, and over GRPO by 1.6 and 3.1 points, respectively. SPGRPO outperforms standard Shapley on six of the seven datasets for each model, supporting the effectiveness of sequential modeling on both a newer Qwen model and a different model family.
To assess generalization across models, we extended our ablation study to these two backbones, using standard Shapley as the direct ablation of sequential dependencies and GRPO and IGPO as additional outcome-level and step-level reward baselines, respectively. The detailed results are reported below.
| Method (Qwen3-4B) | NQ | TriviaQA | PopQA | HotpotQA | Wiki | MuSiQue | Bamboogle | Avg. |
|---|---|---|---|---|---|---|---|---|
| GRPO | 43.9 | 63.3 | 46.7 | 42.1 | 40.3 | 16.7 | 72.2 | 46.5 |
| IGPO | 44.0 | 62.7 | 45.8 | 38.9 | 42.1 | 17.2 | 71.4 | 46.0 |
| Standard Shapley | 45.6 | 63.8 | 47.6 | 42.8 | 43.0 | 15.9 | 72.6 | 47.3 |
| SSVPO | 46.1 | 63.5 | 46.9 | 43.2 | 43.9 | 17.0 | 71.4 | 47.4 |
| SPGRPO | 46.5 | 64.4 | 48.2 | 43.9 | 45.0 | 17.9 | 70.6 | 48.1 |
| Method (Llama-3.2-3B) | NQ | TriviaQA | PopQA | HotpotQA | Wiki | MuSiQue | Bamboogle | Avg. |
|---|---|---|---|---|---|---|---|---|
| GRPO | 46.8 | 62.4 | 47.6 | 38.4 | 36.0 | 13.5 | 68.5 | 44.7 |
| IGPO | 46.0 | 63.1 | 45.8 | 41.2 | 40.3 | 15.8 | 67.3 | 45.6 |
| Standard Shapley | 47.3 | 62.9 | 48.5 | 39.7 | 40.2 | 16.3 | 68.5 | 46.2 |
| SSVPO | 47.2 | 62.5 | 47.3 | 41.2 | 38.7 | 16.6 | 69.0 | 46.1 |
| SPGRPO | 48.1 | 63.4 | 48.3 | 43.3 | 42.6 | 17.6 | 71.4 | 47.8 |
Update: We have added SSVPO results on Qwen3-4B and Llama-3.2-3B to the tables above.
Weakness 1&2. The novelty of sequential Shapley is overclaimed. SSVPO (Li et al., 2026) already formulated sequential Shapley-based credit assignment for LLM reasoning, and it should be discussed in the introduction instead of only being cited as a compared method.
Response. We have expanded the Introduction to discuss SSVPO (Li et al., 2026). We would like to clarify that although both methods use Shapley-based credit assignment and share the name “Sequential Shapley,” they differ fundamentally in motivation and methodology, modeling different aspects of sequential reasoning.
SSVPO evaluates a step's contribution across reordered reasoning chains through an insertion MDP. Expressed as a permutation-averaged coalition value, this corresponds to
=\mathbb E_{\pi\sim\mathrm{Unif}(\Pi(S))}[V(\pi(S))],$$ where $\Pi(S)$ denotes the permutations of the steps in $S$. Our method instead constructs $$\widetilde V_{\mathrm{SPGRPO}}(S)=\sum_C P(C\mid S)V(S\cup C),$$ where $P(C\mid S)$ models how retaining preceding searches changes the likelihood of subsequent continuations. These constructions reflect different motivations. SSVPO addresses the dependence of a step's evaluated contribution on the ordering of the input chain and its insertion position, averaging over these alternatives to obtain fair credit beyond any single arrangement. Our motivation is to use sequential dependencies as a basis for credit redistribution: an earlier search should receive credit for increasing the likelihood of useful later evidence, even when its own evidence has little direct answer utility. Through $P(C\mid S)$, we explicitly model this enabling effect and redistribute credit to the preceding searches that facilitate it. Our contribution is this conditional-continuation formulation and its credit-redistribution interpretation.Thank you for the detailed rebuttal and additional experiments. Most of my concerns are solved.
I keep my original rating of 3 and lean towards acceptance.
Question 1. Could the authors empirically quantify M=maxS∣R(S)∣ on actual search trajectories? How does M compare with sequential and standard Shapley credits empirically?
Response. M measures the largest absolute nonlinear coalition residual within a trajectory. Its mean is 0.0791, approximately half the mean absolute standard and sequential Shapley credits (0.167 and 0.158, respectively). We introduce M as a tool for deriving a theoretical bound on Δ; the accuracy of the linear redistribution approximation is directly determined by Δ, rather than by the bound itself. The actual mean absolute per-step Δ is only 0.00560, roughly 1/30 of the mean absolute Shapley credits. These results support the linear redistribution interpretation of Theorem 2 as a close approximation on the evaluated trajectories.
We generated 600 search trajectories on the four multi-hop QA benchmarks. We study the 475 trajectories that executed at least two searches, since single-search trajectories have identically zero residual. For each trajectory, we computed both Shapley values, using p=0.2 for Sequential Shapley.
For each trajectory, we compute the coalition residual and its maximum magnitude:
\qquad M=\max_{S\subseteq N}|R(S)|.$$ We also directly compute the remainder $\Delta$ in Theorem 2, whose bound is given in Corollary 1: $$\Delta=\phi_{\mathrm{seq}}-(I+B)\phi_{\mathrm{standard}}.$$ The table reports $\overline M$ and the mean absolute magnitudes of standard credits, sequential credits, and $\Delta$. For the per-step quantities, we average within each trajectory and then across trajectories. Numerical metrics are rounded to three significant figures. | Group | Trajectories | $\overline M$ | $\overline{\lvert\phi_{\mathrm{standard}}\rvert}$ | $\overline{\lvert\phi_{\mathrm{seq}}\rvert}$ | $\overline{\lvert\Delta\rvert}$ | | ----------------------- | -----------: | --------------: | --------------------------------------------------: | ---------------------------------------------: | --------------------------------: | | All trajectories | 475 | 0.0791 | 0.167 | 0.158 | 0.00560 | | Successful trajectories | 175 | 0.124 | 0.360 | 0.343 | 0.00973 | | Failed trajectories | 300 | 0.0531 | 0.0539 | 0.0495 | 0.00320 | | HotpotQA | 103 | 0.107 | 0.161 | 0.150 | 0.00761 | | 2WikiMultihopQA | 134 | 0.0885 | 0.150 | 0.141 | 0.00607 | | MuSiQue | 121 | 0.0242 | 0.139 | 0.136 | 0.00136 | | Bamboogle | 117 | 0.101 | 0.219 | 0.206 | 0.00769 |