acceptodds
Under review as a conference paper at ICLR 2027

MASHAP-LLM: Contribution-Aware Policy Optimization for Multi-Agent LLM Post-Training

Abstract

Multi-agent large language model (LLM) systems face a credit assignment problem during training. Shared rewards give every agent the same feedback regardless of its contribution, so policy updates have no explicit signal of which agent's behavior to reinforce. Measuring each agent's contribution by rerunning the task with subsets of agents, called coalition rollouts, is costly. We propose MASHAP-LLM, a multi-agent LLM post-training algorithm that rewards each agent according to its contribution and needs coalition rollouts only early in training. MASHAP-LLM builds on proximal policy optimization (PPO) and assigns rewards using Shapley values. At its core is LLM-SHAP, a learned component that reads the task and the agents' trajectories and estimates the coalition values used to compute these contributions. LLM-SHAP is trained on coalition rollouts scored by a verifier during an early fitting phase and then frozen. After freezing, LLM-SHAP scores each coalition from the full-team trajectory with the outputs of absent agents masked, so no smaller team is generated or verified. MASHAP-LLM turns the resulting Shapley credits into per-agent rewards through a rescaling that preserves the total team reward. We evaluate 3B models from the Qwen2.5 family on three math benchmarks and three coding benchmarks. Against the strongest baseline in each of nine settings, MASHAP-LLM improves the average score by 2.51 percentage points with two agents and 4.36 points with three agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM multi-agent systems, reinforcement learning, credit assignment, Shapley values, LLM post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_opKqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MASHAP-LLM trains sequential multi-agent LLM teams with PPO using per-agent rewards. Each agent’s reward is its Shapley value over coalition values, rescaled so the rewards sum to the team reward. Failed tasks give every agent zero.

After a warmup training, a value head is trained by supervised regression on real coalition outcomes. For each task, the team is rerun with some agents left out: a missing agent’s turn is replaced by a fixed “did not participate” message, and the remaining agents regenerate their responses. The verifier scores each rerun, and those scores are the labels.

The experiments use Qwen2.5-3B teams of 2–5 agents. The paper claims average gains of +2.51 (two agents) and +4.36 (three agents) over the strongest of C3, CCPO, SHARP and MARFT-A.

Critical strengths

The paper's quantitative results are comprehensive -- employing various statistical tools including paired seeds, prespecified Bonferroni families, noninferiority margins fixed in advance, and same-information controls). These make the paper’s claims more checkable.

Critical weaknesses
  1. The gains may not come from better credit assignment. The paper’s own results point elsewhere. Policies trained on rewards computed from the estimator’s predicted coalition values do better than policies trained on rewards computed from the true verifier scores (Table 36). Computing Shapley credit directly from verifier scores also does worse, at 49.6 versus 54.4 with three agents (Table 3). The Reasoner also seems to matter little: removing it barely changes how likely the final agent is to give its recorded answer (Table 18). A control that keeps equal allocation and adds LLM-SHAP’s values as a shaping term would better support the paper’s central claim.
  2. Baseline performance and comparative advantage concerns. Under matched GPU-hours, C3 improves by about 4 points while MASHAP stays flat. On MBPP, the SFT Qwen2.5-3B-Instruct team scores 5-12%, which is unusually low.
Questions for authors
  1. Could the authors provide PPO results with LLM-SHAP’s value model as a per-role critic or shaping reward, without Shapley. I think it make stronger the Shapley claim stands.
  2. Why do learned values outperform the oracle they regress to? From Appendix, table 36.
  3. The MBPP and MBPP+ scores look far too low for the SFT Qwen2.5-3B-Instruct team. It scores 5.31% (two agents) and 12.15% (three agents), and no method exceeds 20%. I concern there is a format or harness issue. Could the author provide more details and check on the figures?
  4. Please reconcile the token and GPU-hour confusion above.
  5. Were the main-table baselines run at equal compute? If not, which numbers should readers treat as the headline result?
3 comments

Thank you for the constructive review and for proposing the shaping control. In short: (1) with the learned values fixed, Shapley allocation exceeds LOO credit, additive role credit and the equal-allocation shaping control you proposed. (2) The main tables are not compute-matched. Under a common GPU-hour budget, MASHAP-LLM exceeds the strongest of six baselines by 1.79 points on MATH500 and 0.64 on Codeforces, and the abstract now reports this result. (3) The low MBPP scores came from our harness and are corrected.

Weaknesses

1. Whether the gains come from credit assignment.

The most direct test fixes the learned coalition values and varies only how they become rewards. Shapley allocation exceeds LOO by 1.36/0.49 points and Additive-role by 1.67/0.61 on MATH500/Codeforces, with positive corrected intervals (24/12 paired seeds, Table 40, originally 36). The shaping control agrees, with Full leading in both seeds on all three math benchmarks (Q1, Table 41). Task alignment also matters. Full exceeds Task-shuffled, which changes only the task-to-credit assignment, by 2.08/0.63 points.

The gap in Table 3 (54.4 vs. 49.6) also reflects different starting states and seed sets. In the matched comparison (Table 40), learned values exceed verifier values by 1.45/0.53 points (Q2).

Table 19 (originally 18) measures the likelihood of the same recorded final answer after upstream information is removed. A small change means the recorded answer remains plausible without the Reasoner's message, not that the Reasoner is unimportant. When downstream roles regenerate, removing the Reasoner changes the verifier score on 14.2%/18.8% of two-/three-role MATH episodes, 14.2%/14.6% on CMATH and 82.4%/83.8% on Code, where partial test credit makes changes frequent. With two roles these rates equal VD in Table 28, since the coalition without the Reasoner is the only answer-capable strict coalition. On successful episodes the Reasoner receives a mean share of 0.10–0.13 (Table 30).

2. Compute and baseline performance.

Tables 1, 2 and 32 (originally 29) use each method's own fixed schedule and are not compute-matched. For seed 103 (three roles, MATH), Full used 2,299.7 GPU-h, and the reconstructed costs of C3, CCPO and SHARP are 571.9, 568.6 and 570.7 GPU-h. Fixed-schedule and envelope runs used different cluster conditions, so per-update costs differ (Full's 8,000 fixed updates cost about 640 GPU-h less than envelope rates predict) and cross-protocol ratios are approximate. The closest-cost comparison is Equal, which shares our PPO scaffold and update count at 0.80× our GPU-hours (1,849.2 GPU-h). Full leads it by 8.66 points on MATH500 (54.23 vs. 45.57, 24 paired seeds).

Under the GPU-hour envelope, C3 runs 617 updates instead of 148, about four times as many, and improves from 48.19 to 52.34. MASHAP-LLM completes 6,481 of its 8,000 updates with an essentially unchanged score (54.36 vs. 54.23), indicating a plateau. At 25/50/75/100% of the GPU allowance, Full's margin over the better of C3 and SEPO is −4.42, +0.74, +1.20 and +1.79 on MATH500 and −1.71, +0.33, +0.49 and +0.64 on Codeforces. The early deficit reflects the fitting phase, which costs 654.7 GPU-h on MATH, 27.3% of the 2,400 GPU-h cap (Table 37). The resource-matched headline is given under Q5, and the MBPP correction under Q3.

Questions

Q1. LLM-SHAP values as a critic or shaping reward.

We ran the shaping variant, as it directly tests whether Shapley aggregation adds value beyond the learned values. For MATH seeds 103 and 113, both arms fork from the same frozen-estimator checkpoint and train to update 8,000. The control uses rᵢ = R/3 + mᵢ − mean(m), where mᵢ = v̂(N) − v̂(N without i). It retains the learned values, replaces Shapley allocation and Eq. (4) with equal allocation plus LOO shaping, and permits signed rewards on failures (Table 41, mean ± sample SD over two seeds).

Variant (update 8,000)MATH500 (%)CMATH (%)GSM8K (%)
MASHAP-LLM53.90 ± 1.2785.56 ± 1.2282.07 ± 1.13
Equal + LOO shaping52.50 ± 1.5684.65 ± 1.3581.20 ± 1.29
Paired difference (pp)+1.40 ± 0.28+0.91 ± 0.13+0.87 ± 0.16

Full leads in both seeds on all three benchmarks. With two seeds the result is directional, but the MATH500 gap (+1.40) agrees with the 24-seed comparison against LOO (+1.36, Table 40).

Q2. Why learned values outperform the oracle.

In the matched comparison the gap is 1.45 points on MATH500 and is reproducible (one negative seed of 24). Learned credit stays close to the verifier on average but reduces advantage variance by 19–26%, which indicates noise removal. On Code, the arms also differ in the Eq. (4) fallback on low-reward episodes. Our response to Reviewer tjgA (items 1 to 3) gives the details.

Q3. MBPP and MBPP+ scores.

Your diagnosis was correct: the harness omitted the function interface. With the corrected harness, the untrained SFT team's MBPP-test score rises from 5.31%/12.15% to 76.67%/78.07% (n=2/n=3, mean test-pass fraction). With a named entry point and one public example, single-agent strict pass@1 is 72.80% on MBPP-test and 61.90% on MBPP+, in line with published results. MASHAP-LLM leads C3 in all eight corrected cells (Tables 2 and 9). Our response to Reviewer tjgA (Q3) gives the protocol and full results.

Q4. Token and GPU-hour accounting.

The revision separates generated from processed tokens and adds the MATH stage ledger (Table 37) and budget caps (response to Reviewer tjgA, item 4). Fixed-schedule costs and budget endpoints are separated in Weakness 2, and the caption of Table 43 now notes the difference.

Q5. Equal compute in the main tables.

No. The headline result is the GPU-hour envelope (Table 34, originally 31). Full exceeds the strongest competitor, SEPO, by 1.79 points on MATH500 (54.36 vs. 52.57, corrected interval [0.96, 2.63]) and 0.64 on Codeforces (6.50 vs. 5.86, [0.16, 1.12]), with the highest mean against all six baselines on all four benchmarks. The token envelope gives +1.73 and +0.63 (Table 35). The abstract and introduction now report this result.

AuthorsOpenReview ↗

Follow-up to Reviewer opKq

Thank you again for your review. As the discussion period is closing, we would like to check whether our responses address your concerns. In brief:

  • Shaping control. We ran the control you proposed, equal allocation with LLM-SHAP values as LOO shaping. MASHAP-LLM leads in both seeds on all three math benchmarks (+1.40 on MATH500, Table 41), in line with the 24-seed comparison against LOO (+1.36, Table 40).
  • MBPP. The harness issue you diagnosed is fixed. The untrained SFT team now scores 76.67%/78.07% (n=2/n=3), and MASHAP-LLM leads C3 in all eight corrected cells.
  • Compute. The headline result is now the GPU-hour envelope, where MASHAP-LLM exceeds the strongest of six baselines by 1.79 points on MATH500 and 0.64 on Codeforces. The token tables separate generated from processed tokens, with a stage ledger in Table 37.

Reviewer tjgA has confirmed that the token accounting, MBPP harness and Eq. (4) issues are resolved. We would be grateful if you could consider whether the revision changes your assessment, and we are happy to clarify any remaining point.

Reviewer_tjgAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The MASHAP-LLM framework is introduced in this paper as a post-training approach for role-diverse LLM teams and gives policy update based on contribution of each member. It starts by collecting coalition traces labeled by verifiers, after which LLM-SHAP is trained to predict the coalition values. Once the initial training is done, the estimator is frozen, and it makes predictions for masked views of entire trajectories without generating and verifying more strict-subcoalitions. These predictions are mapped to Shapley credits and normalized to sum up to the total observed team reward and used for policy updates in each role. The framework is evaluated on mathematical reasoning and code evaluation tasks using the Qwen2.5-family 3B models.

Critical strengths
  1. There is strong evidence to back up the utility of the full procedure. In addition to the five-seed tables, Appendix K provides the 24/12-seed mathematical/code comparison, multiplicity adjusted intervals for contrast analysis, and envelopes for GPU-hour and generated-output-token costs. The resource-constrained mean differences reported are less than those in the main tables but still positive.
Critical weaknesses
  1. The learned estimator outperforms its own supervision target, and the reason is not well clarified in the paper. LLM-SHAP learns a regression model to predict verifier-induced coalition values, and its credits are consistent with the verifier's (credit MAE ≈ 0.02-0.03, Table 21). Nevertheless, policies trained based on its predictions (54.44 on MATH500, n=3) outperform those trained based on exact verifier Shapley credits (Verifier CF, 49.60) by 5 points. They also beat Oracle-rerolled. Errors in credits of such magnitude cannot justify the performance gap, and "shaping" is not an explanation. It means that the gain may be achieved due to something else, contradicting the paper's main argument.

Two closely related problems complicate the interpretation of the result.

(i) Oracle-rerolled (53.02) and Verifier CF (49.60) seem to use the same verifier reference, and they differ by 3.4 points, and the reason for the discrepancy is not clearly explained in the paper.

(ii) In Sec. 5.3, it is claimed that the learned vs oracle comparison isolates the problem of value estimation. However, by the information provided in Appendix D.3/C.1, the credits obtained via the verifier may avoid Eq. (4), so there could be an additional difference in reward transformation.

  1. The reported token usage contradicts the stated training procedure. According to Table 11 (batch size 256), Table 10 (maximum of 512 tokens per agent) and Algorithm 2 (3 replies from the team + $\le$5 coalition replies), the total number of generated tokens is bounded as follows: T≤256×512×(3U+5F)T \le 256 \times 512 \times (3U + 5F). For MATH MASHAP (U=6,481U=6,481), this constraint gives an upper limit of $\sim$3.34 billion tokens, well under the reported 5.223 billion (as well as a similar $\sim$40% difference in the token/MATH column). To properly assess the compute comparison, the authors should disclose a per-stage budget ledger and exact budget parameters.
Questions for authors
  1. Even in Table 34 (MATH500, n=3n=3), Task-shuffled (52.15), Label-shuffled (51.56), and Grand-only (52.33) still greatly outperform C3 (48.19) and Equal (~46), which indicates that much of the gain compared to previous baselines might not rely on task-relevant Shapley credit. Could you make it clear about what else makes the contribution and how do you obtain the matched controls for isolating the effect of coalition-aware credit learning?
  2. AT-GRPO and Shared-GRPO have identical numbers of calls and updates under the token budget in two domains that you have shown (MATH: 2.211M/1080; Code: 1.317M/643). Is this pure coincidence, an intentional design choice since both use shared schedules, or an error when logging those statistics? Please give your explanation for these results.
  3. The coding baseline (Table 2) seems surprisingly low. The backbone SFT model yields only 5.31% (n=2) and 12.15% (n=3) on MBPP-test, well below the published performance of Qwen2.5-3B-Instruct. Please explain details about the evaluation protocol.
5 comments

Thank you for the careful review. In short: (1) in the matched comparison, learned credit exceeds oracle credit by 1.45 points on MATH500, and the larger gap to Verifier CF also reflects starting state and seed set. (2) The token table counted processed tokens. The MATH run generated 0.841 B tokens, within your bound, and the revision adds the stage ledger and budget parameters. (3) The low MBPP scores came from our harness. With the corrected harness, the single-agent backbone reaches 72.80% strict pass@1.

Weaknesses

1. Learned values outperform their supervision target.

The 5-point gap (54.44 vs. 49.60 for Verifier CF) combines several differences (item 2). In the matched comparison, Learned-rerolled and Oracle-rerolled share rerolled inputs and the training prefix. Learned credit leads by 1.45 points on MATH500 (54.47 vs. 53.02, corrected interval [0.69, 2.21]) and 0.53 on Codeforces (6.51 vs. 5.98, [0.21, 0.85]), with one negative seed of 24 on MATH500 (Table 40, originally 36).

This gap is not a bias in mean credit. Raw credit MAE against the verifier reference is 0.022–0.034 across six settings (Table 22, originally 21), and throughout three-role MATH training learned role means remain within 0.006 of the verifier reference (Table 24, new). On successful MATH and CMATH episodes, LLM-SHAP assigned the final role over 90% of the reward exactly when that role alone reproduced the team outcome.

Two factors are consistent with our data (Appendix K.4). First, within-batch variance of per-role advantages is 19–26% lower under learned credit. The oracle scores each coalition with a single greedy reroll, whose outcome can flip under small context changes, whereas LLM-SHAP regresses toward the expected value over similar contexts. We do not claim that this effect has been isolated. Second, on Code, Eq. (4) treats the two arms differently on low-reward episodes (item 3).

The same-value controls indicate that the gain depends on coalition credit. With the learned values fixed, Shapley aggregation exceeds leave-one-out (LOO) credit by 1.36/0.49 points and additive role credit by 1.67/0.61 on MATH500/Codeforces, and task-aligned credit exceeds task-shuffled credit by 2.08/0.63. All Bonferroni-corrected intervals exclude zero (24/12 paired seeds, Table 40). Task-shuffled preserves the share distribution yet performs worse, so the benefit is not merely smoother rewards. We claim that LLM-SHAP is a better training signal, not a more accurate definition of the reference game.

2. Oracle-rerolled versus Verifier CF.

Oracle-rerolled inherits the trained policy, critic, optimizer and data cursor at freeze. Verifier CF starts from SFT, uses verifier credit for all 8,000 MATH updates and runs on the five main seeds instead of the 24 paired seeds. The 3.42-point difference (53.02 vs. 49.60) therefore reflects starting state, optimizer history, seed set and the duration of verifier-credit training. Appendix K.4 now states this.

3. What the oracle comparison holds fixed.

Both arms apply Eq. (4). On MATH, the comparison isolates value estimation. Exact Shapley credit sums to R = 1 on successes and was zero on all audited failures, so Eq. (4) leaves it unchanged. On Code, isolation is incomplete. For exact credit the denominator equals R, so on episodes with 0 < R < τ_floor = 0.05 the oracle always falls back to equal allocation, whereas learned credit falls back only when its own denominator is below the floor or negative. Per Appendix K.4, the learned arm falls back on 83.32% of these episodes, and the overall fallback rates are similar (31.49% vs. 29.98%). Section 5.3 now notes this.

4. Token accounting.

You are correct. Table 36 (originally 33) labeled processed tokens as generated output. In the MATH GPU-envelope run, the 5.223 B total comprises 0.841264 B generated tokens plus prompts, role histories and estimator inputs, which is within your bound. Fitting ended at update 807 of 6,481. Each update uses 256 tasks, with one team verifier call per task and three additional coalition calls per task during fitting. Per-role generation limits are 512 tokens on MATH and 1,024 on Code. The stage ledger is now Table 37.

StageGenerated tokens (B)GPU-hVerifier calls (M)
Team rollouts, fitting0.087388185.70.206592
Coalition rerolls, fitting0.139450338.10.619776
Estimator fitting0130.90
Team rollouts, frozen0.6144261047.61.452544
Estimator inference, frozen0154.70
PPO policy and critic updates0523.80
Total0.8412642380.82.278912

The token envelope (Table 35, originally 32) is relabeled as a processed-token budget, and Tables 4 and 36 and Section 5.4 are corrected. Estimated generated output at its endpoints ranges across methods from 0.849 to 1.293 B on MATH and from 0.875 to 1.281 B on Code.

The caps were 2,400 GPU-h or 6.000 B processed tokens for MATH and 3,200 GPU-h or 4.500 B for Code, applied as separate conditions. Before each update, a run compared its remaining balance with its method-specific reserve, admitted the update at equality and stopped once the balance fell below the reserve. Appendix K.3 lists the reserves.

Questions

Q1. Why the shuffled and grand-only controls exceed C3 and Equal.

These controls are not credit-free. All three inherit Full's policy and critic at freeze, so their first 13–15% of updates used task-aligned learned credit (for example, freeze at update 1,012 of 8,000 for MATH seed 103). After freeze, Task-shuffled permutes whole share vectors among positive-reward episodes within each batch, scales them by the recipient's reward and assigns zero to failures. Label-shuffled and Grand-only use value heads fitted in the same phase on shuffled coalition labels or grand-coalition-only targets. Their advantage over C3 and Equal thus partly reflects the common prefix and the role-share distribution. The matched test is Full versus each control after the prefix: 2.08/0.63 (Task-shuffled), 2.68/0.80 (Label-shuffled) and 1.90/0.56 (Grand-only) on MATH500/Codeforces, all with positive corrected intervals (Table 40). For reference, MATH500/Codeforces scores are 45.57/4.41 for Equal, 47.28/4.51 for Role prior (both trained from SFT), 52.15/5.83 for Task-shuffled and 54.23/6.46 for Full (Tables 38 and 39). Appendix K.4 now describes these protocols.

Q2. Identical GRPO rows.

This follows from the shared schedule and is not a logging error. Both methods use the same processed-token stopping rule, the same reserve and 2,048 requests per update, so both stopped at step 1,080 on MATH and 643 on Codeforces. For example, MATH AT-GRPO seed 103 had 32,737,485 tokens left before step 1,080 and 27,207,121 after it, below the 30 M reserve. After cache hits, mean execution counts are 2.211 M and 1.317 M. GPU-hours differ (Table 36). Appendix K.3 now lists the reserves.

Q3. Low coding baseline.

The low scores were caused by our harness. Its prompts requested a stdin/stdout program and omitted the function signature and public examples. We now provide the named entry point, a function-specific prompt for each role and one public example, and the extractor selects the requested function (Appendix E.3.1). With greedy decoding, EvalPlus 0.3.1 and a 2.0-second limit per test, single-agent Qwen2.5-3B-Instruct reaches strict pass@1 of 72.80% on MBPP-test (500 tasks) and 61.90% on MBPP+. On the 378 EvalPlus base tasks it passes 72.75%, close to the 73.0%/62.2% reported for the same model in BatCoder (Xu et al., 2026, arXiv:2602.02554, Table 1). The untrained SFT team's MBPP-test score rises from 5.31%/12.15% to 76.67%/78.07% (n=2/n=3, mean test-pass fraction).

All trained team outputs were re-evaluated with the unchanged Codeforces checkpoints and the five main seeds (mean ± sample SD, %). Macro is the mean fraction of tests passed (Table 2), and strict is pass@1 (Table 9, new). C3 is the strongest baseline in all eight cells.

MethodMBPP-test, n=2MBPP+, n=2MBPP-test, n=3MBPP+, n=3
C3, macro78.07 ± 0.4371.47 ± 0.6078.99 ± 0.3071.79 ± 0.67
MASHAP-LLM, macro81.03 ± 0.9275.14 ± 0.5682.99 ± 0.5877.63 ± 1.16
C3, strict74.48 ± 0.5464.55 ± 0.8675.64 ± 0.3864.39 ± 0.61
MASHAP-LLM, strict77.24 ± 0.9867.83 ± 0.8979.08 ± 0.9069.58 ± 1.64

MASHAP-LLM exceeds C3 in all eight cells, by 2.96/3.68 (macro) and 2.76/3.28 (strict) on MBPP-test/MBPP+ with two roles, and by 4.00/5.84 and 3.44/5.19 with three. All trained teams exceed the single-agent backbone (75.87/68.29 macro, 72.80/61.90 strict). The untrained SFT team scores 76.67/70.80 (n=2) and 78.07/71.60 (n=3) on MBPP-test and 71.47/64.02 and 75.35/66.93 on MBPP+ (macro/strict). The SFT MBPP cells in Table 2 show the single-agent reference. The nine-setting averages become +2.69/+4.48 points (previously +2.51/+4.36).

Codeforces is unaffected by the fix. Of its 393 held-out tasks, 150 are rated 1600 or higher, and three-role Full's mean test-pass fraction falls from 17.07% below rating 1200 to 1.11% at 2000 or higher. Fully solved problems are rare for every method (two to three of 393 per run), so the paper reports mean test-pass fraction.

Which numbers to use. Table 34 is the resource-matched comparison, and Table 40 reports the paired component comparisons.

Reviewer_6vz1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a multi-agent post-training algorithm for large language models, in which LLMs are trained with additional team-credit assignment heuristics. These heuristics take the form of Shapley values, which are intended to estimate the contribution of individual agents to the team reward. The resulting method, MASHAP-LLM, improves performance over baseline methods across various math and coding benchmarks in 2 and 3 agent settings.

Critical strengths
  • Team-based credit assignment is an open problem in multi-agent reinforcement learning, and especially so with LLM-based agents. Given the large numbers of agent-based teams, this is a timely and practical issue to address.

  • The usage of shapley values as a credit assignment heuristic is interesting, and the early training/freezing is a practical solution to the expensive computational problem of using shapley values.

  • Performance improvements are quite positive, and show reasonable gains across all benchmarks (particularly math).

Critical weaknesses
  • Shapley values are used as a heuristic for assigning credit. This may not necessarily be an accurate heuristic, e.g. removing an agent changes the inputs to other agents, credit assignment is ambiguous, reward normalization is non-linear and can affect attribution, etc. The paper would benefit from some quantitative/qualitative analysis showing whether the Shapley values are actually correlated with true credit.

  • The rescaling rule gives no credit for failure. This means specific agents are not penalized if they directly cause a team failure. Is there a reason why it is desirable to only give credit for success but not failure?

  • The experiments are limited to 2 and 3 agent settings, which are quite modest. It is likely that shapely-value based credit assignment becomes more difficult (and expensive) in larger teams. Smaller results are included in Table 5, but these lack the same analysis as in 2/3 agent teams.

Questions for authors
  1. Is there anything to show that shapley values are faithful to actual team credit?

  2. Is there any reason to not perform credit assignment for team failures?

2 comments

Thank you for the positive assessment. In short: (1) learned credit agrees with a verifier-scored coalition game and with the quality of intermediate outputs. (2) Zero reward on failures loses no information in our settings and still yields role-specific updates. (3) Four- and five-role teams have the same paired analysis, and cost per update grows roughly linearly.

Weaknesses

1. Faithfulness of Shapley credit.

LLM teams have no ground-truth causal contribution, so we test the credit against two references. Your concern that removing an agent changes other agents' inputs is addressed by the reference game. Each strict coalition containing an answer-capable role is re-executed, so downstream agents regenerate their outputs. Coalitions without such a role score zero. Against this reference, value MAE is 0.051–0.055 and raw credit MAE 0.022–0.034 across six settings (Table 22), and learned role means remain within 0.006 of the reference throughout training (Table 24).

Actor answers are scored by exact match and Coder programs by their unit tests (Table 31, new).

Role (n=3)Positive subsetCorrect shareWrong shareCorrect / wrongSpearman ρ
Actor (MATH)All positive0.0933310.040140411 / 3660.585
Actor (MATH)Reviewer alone fails0.2129470.187956178 / 780.212
Coder (Code)All positive0.1999000.0832116 / 8100.777
Coder (Code)Reviewer alone failsn/a0.0834280 / 2460.772

Correct intermediates receive more credit than wrong ones wherever both occur (ratios 1.13–2.40), and credit is rank-correlated with quality, including the subset where the Reviewer alone fails. Appendix I.6.1 adds three MATH transcripts: a Reviewer repair receiving 94% of the credit, a decisive Reasoner plan receiving 45%, and a task requiring all three roles with near-equal credit.

On normalization, the rescaling branch of Eq. (4) multiplies all credits by one positive factor, preserving ratios and ordering. The fallback assigns equal shares. In the MATH diagnostic sets it occurred only on failed episodes, where all rewards are zero under either branch. On Code it also affects a minority of low-reward episodes (Appendix K.4).

2. No credit on failures.

First, this loses no information relative to the reference game, as none of the 1,348 failed diagnostic episodes had a nonzero exact Shapley vector (Appendix C.1). Second, failures still yield role-specific updates. Each role's advantage is −Vω(cᵢ), so roles the critic expects to be decisive, such as the Reviewer (mean share ≈0.82 on successes, Table 30), are pushed down most. This reflects expected rather than realized responsibility. Signed failure credit, for example from checks that detect an overwritten correct answer, is a natural extension, now noted in Section 7.

3. Only two and three agents.

Table 33 (originally 30) provides the same paired analysis for four and five roles (24 MATH and 12 Code seeds). They exceed C3 by 3.49/3.80 points on average, and all eight corrected intervals are positive. With shared-permutation sampling (K = 20), MATH GPU-hours per update are 1.15×, 1.24×, 1.34× and 1.45× those of Equal for two to five roles. Value MAE is 0.048–0.058 at four/five roles versus 0.051–0.055 at two/three, and raw credit MAE is 0.013–0.016 (Table 25). Role configurations change with team size.

AuthorsOpenReview ↗

Follow-up to Reviewer 6vz1

Thank you again for your positive review. As the discussion period is closing, we would like to check whether our response answers your questions. In brief, learned credit agrees with the verifier-scored coalition game and ranks correct intermediate outputs above wrong ones (Table 31, new). Zero reward on failures loses no information in our settings, and signed failure credit is now noted in Section 7. Four- and five-role teams have the same paired analysis, with cost per update growing roughly linearly (Table 33). We are happy to clarify any remaining point.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank all three reviewers. Changed text in the revised PDF is marked in blue. Table numbers in our responses refer to the revised PDF, with the original number in parentheses where a reviewer cited a table. The main text stays within the 9-page limit.

Main text. The abstract, introduction and conclusion now report the resource-matched result next to the fixed-schedule averages. Under a common GPU-hour budget, MASHAP-LLM exceeds the strongest of six baselines by 1.79 points on MATH500 and 0.64 on Codeforces (opKq, tjgA). After the MBPP correction, the nine-setting averages are +2.69/+4.48 points (tjgA, opKq). Section 5.3 notes that on Code the learned-versus-oracle comparison also reflects the Eq. (4) threshold (tjgA). Section 5.4 states that the fixed-schedule tables are not compute-matched and relabels the token budget as processed tokens (tjgA, opKq). Section 7 names signed failure credit as a natural extension (6vz1).

Code evaluation. Table 2 reports MBPP with the corrected harness, Table 9 adds strict pass@1, and Appendix E.3.1 describes the protocol. Obsolete MBPP columns are removed from the appendix tables (tjgA, opKq).

Appendix. I.2 adds credit across training (Table 24), and I.6.1 adds intermediate-output quality and three transcripts (Table 31) (6vz1, tjgA). K.3 adds the generated-token count, the MATH stage ledger (Table 37), the budget caps and the method-specific reserves (tjgA, opKq). K.4 adds the control protocols, the difference between Oracle-rerolled and Verifier CF, the Eq. (4) fallback rates, the advantage-variance result and the shaping control (Table 41) (tjgA, opKq). The caption of Table 43 explains GPU-hour accounting across protocols (opKq).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.