acceptodds
Under review as a conference paper at ICLR 2027

Who Deserves the Reward? Shapley Credit Optimization for Multi-Agent LLMs

Abstract

Integrating Large Language Models (LLMs) with external tools via multi-agent systems offers a promising paradigm for decomposing and solving complex problems. However, training multi-agent LLM systems with tool use remains challenging due to the credit assignment problem, as it is often unclear which agent interactions contribute to the success or failure of decision trajectories. Existing LLM agent training methods typically rely on globally broadcast rewards or augment them with heuristic or LLM-judge-based local feedback, failing to capture marginal contributions from individual agents and leading to inefficient reinforcement learning. To address these limitations, we introduce the Shapley-based Hierarchical Attribution for Reinforcement Policy (SHARP), a credit-aware optimization framework tailored to multi-agent LLM reinforcement learning with tool use. SHARP stabilizes training by normalizing agent-specific advantages across trajectory groups and decomposes the reward into three components: a global broadcast accuracy reward for task alignment, a Monte Carlo Shapley based marginal credit reward for agent-level attribution, and a tool format reward for valid tool invocation. Extensive experiments across real-world LLM agent benchmarks demonstrate that SHARP significantly outperforms recent baselines, achieving average match improvements of 35.01% and 24.97% over single-agent and multi-agent LLM agent approaches, respectively. The implementation is available at https://anonymous.4open.science/r/SHARP-DDB4.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, credit assignment, preference optimization, shapley value

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_foxiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SHARP for credit assignment in tool-using multi-agent LLMs. The method combines the accuracy reward, the Monte Carlo Shapley marginal credit reward, and the tool format reward, and optimizes each agent based on agent-specific group-relative advantage. To compute marginal credit, the authors sample agent coalitions, remove trajectory segments generated by agents not in the coalition, and perform counterfactual rerollouts for the affected downstream agents; the difference in the correctness of the final answer before and after including a particular agent represents its marginal contribution. The paper was trained on MuSiQue and evaluated on MuSiQue, GAIA-text, WebWalkerQA, FRAMES, and DocMath-Eval. Compared to baselines such as MATPO, SHARP reports performance improvements.

Critical strengths
  1. The method integrates counterfactual coalition evaluation into the training process of tool-using multi-agent LLMs, addresses a clearly defined problem, and requires no additional LLM judge. Table 2 indicates that this reward plays a practical role in the current system.
  2. The evaluation covers multiple tasks and reports the relationship between sampling budget and training overhead. In Table 1, the average score for the Qwen3-8B model improved from 30.74 under MATPO to 35.63 under SHARP. These results support the method’s practical potential.
Critical weaknesses
  • The core attribution claim lacks independent verification. Table 2 shows that the marginal credit reward improves accuracy, but it does not test whether per-agent credit is accurate. Furthermore, Figure 4 defines useful and harmful agents based on Shapley credit itself, and thus is insufficient to independently support the paper’s conclusion regarding improved coordination.
  • Insufficient necessity and novelty. Overall, SHARP is a direct combination of existing reward components and group-relative optimization, with the primary increment being the introduction of Monte Carlo Shapley credit. This increment is relatively limited, and the experiments only validated the benefits of adding marginal rewards without comparing the method to simple credit approaches with comparable costs. Therefore, the unique benefits of the Shapley design remain unclear.
Questions for authors
  1. Given the same backbone, workflow, and counterfactual evaluation budget, what are the accuracy and training overhead of Shapley credit compared to leave-one-out credit and other simple local credits?
  2. Do the signs and rankings of per-agent credit align with the changes in success rates caused by online agent removal with multiple random seeds on the held-out task? Can the “useful/harmful” ratio shown in Figure 4 be reproduced using independent metrics?
  3. After removing a planner or an intermediate agent, does counterfactual rerollout preserve the original call path? How large is the variance in credit when the same coalition is generated repeatedly?
4 comments

Q1. How does SHARP compare with simpler credit rules ?

Table 2 measures the effect of adding marginal credit. To test the choice of credit rule, we compare SHARP with full team leave one out (LOO), a fixed response difference reward, and uniform coalition weighting. Let vi(S)v_i(S) denote the expected terminal accuracy for a coalition SS of calls from trajectory ii. LOO measures vi(Mi)−vi(Mi∖{m})v_i(\mathcal M_i)-v_i(\mathcal M_i\setminus\{m\}) for the target call mm, where Mi\mathcal M_i is the full set of calls. The fixed response baseline replaces the target call's output with a preset response and reuses the original downstream steps. Uniform weighting averages the marginal effect of adding mm with equal weight per subset. SHARP gives each coalition size equal total weight, as in Eq. (12).

The runs used the same backbone and workflow, with eight NVIDIA A100 GPUs for 180 updates each. We kept end to end training costs as close as practicable and estimate GPU hours for each method from the mean step time.

Credit ruleMuSiQue (%)GAIA-text (%)WebWalkerQA (%)FRAMES (%)Training GPU hours
Full team LOO50.7634.188.5037.29435.2
Difference reward with a fixed response47.2934.428.8535.89418.0
Uniform coalition weighting46.0933.777.4336.23487.2
Shapley weighting (SHARP)54.5837.6312.8537.48450.8

Full team LOO measures a call’s marginal contribution only when all other calls are present, so it cannot capture how that contribution varies across different sets of collaborators. The fixed response baseline reuses downstream steps, which limits how it captures changes in later searches or reasoning. Uniform weighting evaluates multiple collaborator sets and puts more total weight on sizes with more subsets. SHARP reruns affected downstream calls and gives each coalition size equal total weight, so a call can receive credit for being useful with fewer collaborators even when it is redundant in the full team.


Q2. How stable are the credit estimates, and how do they compare with online removal?

Stability across random seeds. We evaluated 315 subagent calls on MuSiQue. We fixed the reference trajectories and sampled coalitions and recomputed Shapley credit using three random seeds. The three estimates agreed in sign for 282 calls (89.5%). Across the three seed pairs, Spearman rank correlations ranged from 0.709 to 0.757, with a mean of 0.736.

Variance when repeating a fixed coalition. For each of 589 target call and collaborator set pairs (m,S)(m,S), we evaluated SS and S∪{m}S\cup\{m\} under each of the three seeds and computed the marginal credit Yb(S∪{m})−Yb(S)Y_b(S\cup\{m\})-Y_b(S), where YbY_b is the terminal accuracy score. We computed the sample variance of the three differences for each pair. The three differences were identical for 545 of 589 pairs (92.5%). After averaging across the sampled coalitions with Shapley weights, the mean variance across seeds was 0.01693 over the 315 calls.

Independent evaluation with online LOO. Intermediate subagent calls have no ground truth labels indicating whether they help or harm task success. Asking an LLM judge to assess intermediate outputs would introduce subjective judgments and could make the labels sensitive to the judge and its prompt. We therefore re-evaluate subagent credit for all three systems using online LOO. We classify each call as helpful, harmful, or having no observed effect based on how removing it changes final task success. This gives us a way to test whether the coordination patterns in Fig. 4 hold under a criterion independent of Shapley credit.

ModelHarmfulHelpful
Hierarchical MAS8.14%9.19%
MATPO5.71%11.43%
SHARP4.78%14.24%

Under online LOO, SHARP had the highest proportion of helpful calls and the lowest proportion of harmful calls. This shows that SHARP’s advantage also holds under an evaluation based on final task outcomes, independent of Shapley credit.

Q3. What happens to downstream calls after an intervention?

The dependency structure in Sec. 4.3 and Appendix C identifies which downstream executions used information from the removed segment. These dependencies arise primarily because workers execute subtasks assigned by the planner and rely on the instructions and context it provides. In full downstream rerollout, removing a planner segment or an intermediate call invalidates those executions. We discard the affected outputs and execute the eligible downstream steps again under the modified context. This allows the new execution to adapt its search, reasoning, and subsequent calls.

Comparison with fixed segment replacement. The fixed response baseline substitutes a preset segment and reuses the original intermediate steps. If a retained step relied on the original segment, its output can still carry information that the intervention was intended to remove. Regenerating the affected steps from the edited context addresses this dependency.

MethodMuSiQue (%)GAIA-text (%)WebWalkerQA (%)FRAMES (%)
Fixed segment replacement47.2934.428.8535.89
SHARP54.5837.6312.8537.48

SHARP achieved higher accuracy on all four benchmarks.


We will incorporate your valuable feedback into the revised PDF and upload the updated version shortly.

Thank the authors for their additional experiments, which addressed some of my questions. However, I still believe the innovation is limited and insufficient to meet the acceptance criteria of ICLR. Thus, I will maintain the initial score.

Thank you for reviewing our paper and sharing your concerns about its novelty. We would like to explain the motivation behind SHARP and the contribution we aim to make.LLM agents are developing at an extraordinary pace, bringing new forms of collaboration and new learning challenges. Credit assignment remains an important, incompletely resolved problem. Alongside practical solutions, we believe the field needs clear formal models and interpretable approaches grounded in theory. Connecting these emerging problems with the knowledge accumulated in cooperative game theory and multiagent learning is a direction we consider deeply worthwhile.

SHARP grew out of our experience in collective optimization and multiagent cooperation. We sought to bring that experience into LLM agent learning through a simple and effective approach with an explicit mathematical foundation that others can examine and build upon. We hope to be bridge builders between these research communities. For researchers in traditional optimization, LLM agents offer new questions and opportunities to extend established ideas. For researchers in LLM agents, those ideas offer tools for organizing problems and explaining mechanisms. We hope to help knowledge travel in both directions, giving researchers with different backgrounds a shared language and new ways to contribute.

This connection matters to us because a rapidly advancing field benefits from continuity with earlier research, while its emerging challenges can also inspire established disciplines. Helping these communities learn from one another is a contribution we sincerely wish to make. We respect the importance of originality and hope SHARP can also be considered for the problem formulation and concrete modeling work involved in building this connection, alongside the familiarity of its foundations. We would sincerely appreciate your reconsideration of its contribution from this perspective.

Reviewer_Xbs8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors tries to addres how should reinforcement learning assign credit to individual agents in a tool-using multi-agent LLM system? To do so, the authors propose the SHARP, a reinforcement learning framework combining final-answer correctness, agent-specific marginal credit, and tool-call format rewards. It estimates marginal credit using Monte Carlo Shapley values: sample groups of agents and compare outcomes with and without a particular agent. Experiments show that adding SHARP can help the multi-agent system performs better.

Critical strengths

The authors have designed a concrete adaptation of Shapley attribution, Monte Carlo approximation, counterfactual credit to dependent LLM/tool trajectories and the integrated training procedure.

Critical weaknesses

The experiments do not establish why Shapley attribution is the right additional mechanism. The key novelty is coalition-dependent marginal credit, but the main ablation removes marginal credit entirely. That establishes the usefulness of the added signal within the training recipe, not the necessity of averaging over Shapley-weighted coalitions. A matched comparison with full-team leave-one-out credit, another difference-reward baseline, and alternative coalition weighting is needed. These should hold architecture, reward terms, normalization, and computation as constant as practicable. A favorable comparison would make the central contribution substantially more persuasive.

Questions for authors

How do we justify the usage of Shapley attribution in training multi-agent system?

1 comment

Q1. Why use Shapley attribution in training?

Thank you for asking what Shapley attribution adds to training. The ablation in Table 2 measures the benefit of adding marginal credit. To evaluate the credit rule itself, we consider four methods: full team leave one out (LOO), a difference reward with a fixed response, uniform coalition weighting, and Shapley weighting.

Let Mi\mathcal M_i be the calls in trajectory ii, mm the target call, and n=∣Mi∣−1n=|\mathcal M_i|-1 the number of other calls. Let vi(S)v_i(S) denote the expected terminal accuracy in the counterfactual with coalition SS, and write Δi,m(S)=vi(S∪{m})−vi(S)\Delta_{i,m}(S)=v_i(S\cup\{m\})-v_i(S).

Full team LOO. Use Δi,m(Mi∖{m})\Delta_{i,m}(\mathcal M_i\setminus\{m\}), the effect of removing the target call from the full team. Difference reward with a fixed response. Compare the observed outcome with the outcome under a fixed reference response [1]. For each target subagent call, retain the original prefix and the call's position and message format. Replace its output with a prespecified response, such as “No additional evidence to report.” Regenerate the affected downstream steps and use the same terminal accuracy scorer. If viref(m)v_i^{\mathrm{ref}}(m) denotes the expected terminal accuracy after replacement, the credit is $ D^{\mathrm{ref}}_{i,m}=v_i(\mathcal M_i)-v_i^{\mathrm{ref}}(m). $ Uniform coalition weighting. Give every eligible subset S⊆Mi∖{m}S\subseteq\mathcal M_i\setminus\{m\} weight 2−n2^{-n}. This is equivalent to sampling each other call independently with inclusion probability 1/21/2 and averaging the sampled marginal effects. Shapley weighting (SHARP). Following Eq. (12), give each coalition size the same total weight and divide that weight equally among the subsets of that size. The weight of subset SS is 1/[(n+1)(n∣S∣)]1/[(n+1)\binom{n}{|S|}].

We compare the rules with shared backbone, training data and workflow, broadcast accuracy and tool format rewards and their coefficients. We report training GPU hours for each run.

Credit ruleMuSiQue (%)GAIA-text (%)WebWalkerQA (%)FRAMES (%)Training GPU hours
Full team LOO50.7634.188.5037.29435.2
Difference reward with a fixed response47.2934.428.8535.89418.0
Uniform coalition weighting46.0933.777.4336.23487.2
Shapley weighting (SHARP)54.5837.6312.8537.48450.8

SHARP achieved the highest accuracy on all four benchmarks. All experiments used eight NVIDIA A100 GPUs, and we kept training GPU hours as comparable as possible across methods.

Full team LOO measures a call’s contribution when all other calls are present, so it can miss evidence that is redundant in the full team but useful in smaller coalitions. The fixed response baseline replaces the target call’s output and reuses the original downstream steps. It therefore misses how later calls might change their searches or reasoning in response to the replacement. Uniform coalition weighting considers multiple collaborator sets, with more total weight on intermediate sizes because they contain more subsets. SHARP regenerates the affected downstream steps after each intervention and averages the resulting marginal contributions across collaborator sets, giving each coalition size equal total weight. Its credit can therefore reflect both how later calls respond to the intervention and how the target call’s contribution varies with the collaborators present.


We will incorporate your valuable feedback into the revised PDF and upload the updated version shortly.

Reviewer_WZn6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SHARP, a reinforcement learning framework for tool-using multi-agent LLMs. It combines a broadcast outcome reward, Monte Carlo Shapley credit, and a tool-format reward with agent-specific group-relative optimization. The reported benchmark improvements are promising. However, the current manuscript leaves important aspects of the attribution procedure underspecified, and the experiments do not sufficiently establish that Shapley-specific credit assignment explains the gains.

Critical strengths
  1. The paper addresses credit assignment in tool-using multi-agent LLMs, where a shared outcome reward cannot distinguish individual agents’ contributions.

  2. Combining global task success, marginal contribution, and tool-format validity provides an intuitive way to balance collective performance with agent-specific feedback.

  3. The experiments cover multiple benchmarks and include comparisons with single-agent and multi-agent baselines, five-run statistics for the main results, reward ablations, and a sampling-budget analysis.

Critical weaknesses
  1. The theoretical justification is limited to showing that the Monte Carlo estimator is unbiased for the chosen coalition game. This does not establish that the counterfactual construction faithfully captures agent contributions or that incorporating these credits into the reward preserves alignment with the original task objective.

  2. The proposed method and experimental analysis are not clearly connected. The formulation assigns credit and normalizes advantages by agent identity, while the coordination experiments interpret credit at the level of individual subagent calls. The manuscript does not explain how these units correspond or how agent identities are aligned across sampled trajectories.

  3. The experiments do not sufficiently validate the central contribution. Comparisons with simpler leave-one-out or difference rewards under matched computation budgets are missing, and the reward ablation only establishes the usefulness of an additional feedback component. Consequently, the results do not demonstrate that Shapley-based attribution is necessary for the reported gains.

Questions for authors
  1. Beyond the unbiasedness of the Monte Carlo estimator, what theoretical justification supports using the estimated Shapley credits in the policy update? Does the modified reward preserve alignment with the original task objective?

  2. How does the agent identity used for credit assignment and advantage normalization correspond to the individual subagent calls evaluated in the coordination experiments? How are repeated calls and identities across trajectories handled?

  3. Can the authors compare Shapley credit against simpler leave-one-out or difference rewards under matched computation budgets to establish whether the gains come from Shapley attribution specifically?

4 comments

Q1. Why use Shapley credit in the policy update, and how does it align with terminal accuracy?

Thank you for raising this question. SHARP uses the original terminal correctness criterion to measure the effects of individual calls and uses Shapley credit to allocate the resulting gains across calls.

Counterfactual construction and credit assignment. As described in Sec. 4.3 and Appendix C, we replace excluded calls with null segments and regenerate affected downstream calls that belong to the coalition under the modified context. We score both branches using the same terminal correctness criterion as the original task. The resulting difference measures how retaining a call affects final correctness, including through changes in downstream execution, relative to the specified absence baseline.

Shapley attribution averages these marginal effects across collaborator sets. For the resulting coalition game, a call with zero marginal effect in every coalition receives zero credit, calls with equivalent marginal effects receive equal credit, and the credits sum to the gain of the full coalition over the empty coalition [1]. These properties motivate using Shapley credit as a training signal for each call's action sequence alongside the broadcast accuracy reward, as in Eqs. (14)–(17). In cooperative multi-agent reinforcement learning, Li et al. [2] used counterfactual Shapley credits to guide individual policy updates. Wang et al. [3] developed a theoretical framework for decomposing team value into individual agent values using Markov Shapley values.

Alignment with terminal accuracy. For a reference trajectory τi\tau_i, let rir_i be its binary terminal correctness, ni=∣Mi∣n_i=|\mathcal{M}_i| its number of calls, and fˉi\bar{f}_i its mean format reward. Let vi(S)v_i(S) denote expected terminal correctness under coalition SS. Under the construction in Appendix C, the full coalition retains the reference trajectory, so vi(Mi)=riv_i(\mathcal{M}_i)=r_i. Shapley efficiency then gives $ \sum_{m\in\mathcal{M}_i}\phi_{i,m}=r_i-v_i(\varnothing). $

Taking the expectation over Monte Carlo credit estimation in Eq. (10), the expected mean reward across calls is

Ri=EMC[1ni∑m∈MiRˉi,m]=αri+βni[ri−vi(∅)]+γfˉi.\mathcal{R}_i=\mathbb{E}_{\mathrm{MC}}\left[\frac{1}{n_i}\sum_{m\in\mathcal{M}_i}\bar{R}_{i,m}\right]=\alpha r_i+\frac{\beta}{n_i}\left[r_i-v_i(\varnothing)\right]+\gamma\bar{f}_i.

Since vi(∅)∈[0,1]v_i(\varnothing)\in[0,1] and β≥0\beta\geq 0, the expected credit adjustment is nonnegative for a correct trajectory and nonpositive for an incorrect trajectory. For a correct trajectory (+) and an incorrect trajectory (−) with the same mean format reward,

R+−R−=α+β[1−v+(∅)n++v−(∅)n−]≥α.\mathcal{R}_{+}-\mathcal{R}_{-}=\alpha+\beta\left[\frac{1-v_{+}(\varnothing)}{n_{+}}+\frac{v_{-}(\varnothing)}{n_{-}}\right]\geq\alpha.

Thus, adding Shapley credit preserves or enlarges the expected mean reward gap in favor of a correct final answer, even when the trajectories have different numbers of calls and different empty coalition values. This guarantee applies to rewards before advantage normalization and clipping. In the full training procedure, Table 2 shows that adding marginal credit improves GAIA-text accuracy from 31.65% to 37.63%. We will include this derivation and clarify its scope in the revision.

References

[1] Lloyd S. Shapley. A Value for n-Person Games. Contributions to the Theory of Games, Vol. II, 1953.

[2] Jiahui Li, Kun Kuang, Baoxiang Wang, Furui Liu, Long Chen, Fei Wu, and Jun Xiao. Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning. KDD, 2021.

[3] Jianhong Wang, Yuan Zhang, Yunjie Gu, and Tae-Kyun Kim. SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning. NeurIPS, 2022.


Q2. What does an agent index mean in the method and in Fig. 4?

Section 4.1 defines τt\tau_t in Eq. (5) as the segment generated by “the agent invoked at step tt.” That invocation generates its segment conditioned on ctc_t, which contains the role specification, task state, interaction history, and tool schemas. Algorithm 1 makes the indexing explicit. On trajectory τi\tau_i, it invokes mi,tm_{i,t} with context ci,tc_{i,t}, appends the resulting segment, and adds mi,tm_{i,t} to Mi\mathcal M_i. The role is part of the context, and two invocations of the same worker are distinct members of the coalition game. Equations (14) and (15) assign credit to that invocation. In Sec. 4.4, Eq. (17) uses the action sequence and contexts from the same invocation to compute its policy ratio. Figure 4 uses the sign of each subagent invocation’s credit. The method and the figure thus use the same unit of attribution.

Following Algorithm 1, ii indexes the trajectory and tt indexes the agent invocation within that trajectory. We write Rˉi,t\bar R_{i,t} and A^i,t\hat A_{i,t} for the reward and advantage of mi,tm_{i,t}, respectively. For advantage normalization, we pool the rewards of all invocations from the GG trajectories sampled for the same query qq. Let Iq={(i,t):1≤i≤G, 1≤t≤Ti}\mathcal I_q=\{(i,t):1\le i\le G,\ 1\le t\le T_i\} index those invocations and Nq=∣Iq∣N_q=|\mathcal I_q|. The shared statistics and each invocation's advantage are

μq=1Nq∑(i,t)∈IqRˉi,t,σq=Std⁡((Rˉi,t)(i,t)∈Iq),A^i,t=Rˉi,t−μqσq+δ.\mu_q=\frac{1}{N_q}\sum_{(i,t)\in\mathcal{I}_q}\bar{R}_{i,t},\qquad \sigma_q=\operatorname{Std}\left((\bar{R}_{i,t})_{(i,t)\in\mathcal{I}_q}\right),\qquad \hat{A}_{i,t}=\frac{\bar{R}_{i,t}-\mu_q}{\sigma_q+\delta}.

Each observed invocation contributes one reward to the pool, including repeated invocations of the same worker. Every invocation uses the same μq\mu_q and σq\sigma_q, and its own reward determines its advantage.


Q3. SHARP and LOO

We trained a baseline using leave one out (LOO) credit on the full team. For a target call mm in trajectory τi\tau_i, LOO measures Di,mLOO=vi(Mi)−vi(Mi∖{m})D^{\rm LOO}_{i,m}=v_i(\mathcal M_i)-v_i(\mathcal M_i\setminus\{m\}). This is the marginal effect of removing the call from the full team. SHARP (K=3K=3) estimates marginal credit across multiple collaborator sets. SHARP achieved higher accuracy on all four benchmarks. The largest gains were +4.35 points on WebWalkerQA, +3.82 on MuSiQue, and +3.45 on GAIA-text. The gain on FRAMES was +0.19 points.

Benchmark or training metricLOOSHARPSHARP − LOO
MuSiQue accuracy (%)50.7654.58+3.82 pp
GAIA-text accuracy (%)34.1837.63+3.45 pp
WebWalkerQA accuracy (%)8.5012.85+4.35 pp
FRAMES accuracy (%)37.2937.48+0.19 pp
GPU hours435.2450.8+15.6 (+3.6%)

A call’s marginal contribution can depend on the evidence supplied by the other calls. For example, if two calls retrieve the same decisive fact, removing either one from the full team may leave the answer unchanged. SHARP also evaluates coalitions in which the other call is absent, so the credit can reflect the call’s value when it supplies that evidence on its own. Evaluating contributions across coalitions is particularly important in systems of collaborating LLM agents, where an agent’s contribution can depend on how it works with the others.


We will incorporate your valuable feedback into the revised PDF and upload the updated version shortly.

Thank you for the authors' response. I will maintain my initial score.

Thank you for your thoughtful feedback and constructive suggestions. We sincerely appreciate your time and effort in reviewing our work.

Comments on the paper

1 comment

Summary of Rebuttal

We sincerely thank the Area Chair and all reviewers for their time and feedback. During rebuttal, we conducted additional experiments and clarified the attribution procedure, reward alignment, and invocation indexing. To facilitate the final assessment, we summarize the concerns discussed, the corresponding evidence, and the motivation and contributions of SHARP. The new experiments are included in Appendix F of the revised manuscript and highlighted in blue. Further details are provided in our responses to each reviewer.

Rebuttal Process

The table summarizes additional experiments and clarifications, grouping overlapping concerns. Check marks identify the reviewers who raised each concern.

ConcernWZn6Xbs8foxiLocation in revision / rebuttalMain result or clarification
Comparisons with simpler credit rules and alternative coalition weighting✓✓✓Appendix F.1, Table 3; WZn6 Q3, Xbs8 Q1, foxi Q1SHARP outperforms full team LOO, fixed response, and uniform coalition weighting on all four benchmarks.
Comparable training computation and reported overhead✓✓✓Appendix F.1, Table 3; replies on credit rule comparisonsAll runs use eight A100 GPUs for 180 updates. SHARP uses 450.8 GPU hours, 3.6% more than LOO.
Justification beyond estimator unbiasedness and alignment with terminal correctness✓Rebuttal to WZn6, Q1; the reward definition and counterfactual construction are in Secs. 4.2–4.3 and Appendix CDerived an expected mean reward margin favoring correct answers under the stated assumptions, before advantage normalization and clipping.
Attribution unit, repeated calls, and normalization across trajectories✓Rebuttal to WZn6, Q2, including its continuation; Sec. 4.1 and Algorithm 1 provide the invocation notationCredit is assigned per invocation, including repeated calls. Normalization pools invocation rewards across trajectories for the same query.
Counterfactual execution after removing planner or intermediate outputs✓✓Sec. 4.3, Appendix C; Appendix F.1 for the fixed response control; WZn6 Q1 and foxi Q3Affected downstream outputs are discarded and eligible calls rerun under the modified context. Fixed response reuses recorded downstream steps.
Independent evaluation of helpful and harmful calls✓Appendix F.2, Table 4; foxi Q2Online LOO gives SHARP the lowest harmful (4.78%) and highest helpful (14.24%) call proportions among the three systems.
Credit sign and ranking stability across random seeds✓Appendix F.2; foxi Q2Across three seeds and 315 calls, sign agreement is 89.5% and mean Spearman correlation is 0.736.
Variance when repeating the same coalition evaluation✓Appendix F.2; foxi Q2Marginal differences are identical across three seeds for 92.5% of 589 pairs. Mean variance of aggregated call credits across seeds is 0.01693.

Motivation and Core Contributions

The contribution of SHARP centers on three points:

  1. A formal perspective on credit assignment in LLM agent learning. Multiple calls jointly determine the final outcome, yet their individual contributions can vary substantially. SHARP introduces cooperative game theory to explicitly define contributions across collaboration contexts, providing a principled framework for constructing training signals without additional judgments of intermediate outputs.

  2. Credit assignment for cooperative reinforcement learning in MAS. SHARP estimates marginal contributions across coalitions and integrates these credits into policy optimization alongside the global reward, providing differentiated learning signals that capture cooperation and redundancy among agents.

  3. A practical method supported by targeted controls and independent evaluation. Controlled comparisons examine credit rules under a shared training framework with explicit training costs. Online LOO independently assesses helpful and harmful calls, complementing the Shapley-based training metric and strengthening the empirical evidence for coordination effects.

More broadly, we aim to bridge established credit assignment theory and modern LLM agent learning. This connection brings new problems to traditional optimization research and principled tools to LLM agent research. SHARP represents a concrete step in this direction. We respectfully ask that its contribution be evaluated through its problem formulation, modeling, and empirical evidence, rather than solely through the familiarity of its theoretical foundations.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.