- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
MASHAP-LLM trains sequential multi-agent LLM teams with PPO using per-agent rewards. Each agent’s reward is its Shapley value over coalition values, rescaled so the rewards sum to the team reward. Failed tasks give every agent zero.
After a warmup training, a value head is trained by supervised regression on real coalition outcomes. For each task, the team is rerun with some agents left out: a missing agent’s turn is replaced by a fixed “did not participate” message, and the remaining agents regenerate their responses. The verifier scores each rerun, and those scores are the labels.
The experiments use Qwen2.5-3B teams of 2–5 agents. The paper claims average gains of +2.51 (two agents) and +4.36 (three agents) over the strongest of C3, CCPO, SHARP and MARFT-A.
Critical strengths
The paper's quantitative results are comprehensive -- employing various statistical tools including paired seeds, prespecified Bonferroni families, noninferiority margins fixed in advance, and same-information controls). These make the paper’s claims more checkable.
Critical weaknesses
- The gains may not come from better credit assignment. The paper’s own results point elsewhere. Policies trained on rewards computed from the estimator’s predicted coalition values do better than policies trained on rewards computed from the true verifier scores (Table 36). Computing Shapley credit directly from verifier scores also does worse, at 49.6 versus 54.4 with three agents (Table 3). The Reasoner also seems to matter little: removing it barely changes how likely the final agent is to give its recorded answer (Table 18). A control that keeps equal allocation and adds LLM-SHAP’s values as a shaping term would better support the paper’s central claim.
- Baseline performance and comparative advantage concerns. Under matched GPU-hours, C3 improves by about 4 points while MASHAP stays flat. On MBPP, the SFT Qwen2.5-3B-Instruct team scores 5-12%, which is unusually low.
Questions for authors
- Could the authors provide PPO results with LLM-SHAP’s value model as a per-role critic or shaping reward, without Shapley. I think it make stronger the Shapley claim stands.
- Why do learned values outperform the oracle they regress to? From Appendix, table 36.
- The MBPP and MBPP+ scores look far too low for the SFT Qwen2.5-3B-Instruct team. It scores 5.31% (two agents) and 12.15% (three agents), and no method exceeds 20%. I concern there is a format or harness issue. Could the author provide more details and check on the figures?
- Please reconcile the token and GPU-hour confusion above.
- Were the main-table baselines run at equal compute? If not, which numbers should readers treat as the headline result?
3 comments
Questions
Q1. LLM-SHAP values as a critic or shaping reward.
We ran the shaping variant, as it directly tests whether Shapley aggregation adds value beyond the learned values. For MATH seeds 103 and 113, both arms fork from the same frozen-estimator checkpoint and train to update 8,000. The control uses rᵢ = R/3 + mᵢ − mean(m), where mᵢ = v̂(N) − v̂(N without i). It retains the learned values, replaces Shapley allocation and Eq. (4) with equal allocation plus LOO shaping, and permits signed rewards on failures (Table 41, mean ± sample SD over two seeds).
| Variant (update 8,000) | MATH500 (%) | CMATH (%) | GSM8K (%) |
|---|---|---|---|
| MASHAP-LLM | 53.90 ± 1.27 | 85.56 ± 1.22 | 82.07 ± 1.13 |
| Equal + LOO shaping | 52.50 ± 1.56 | 84.65 ± 1.35 | 81.20 ± 1.29 |
| Paired difference (pp) | +1.40 ± 0.28 | +0.91 ± 0.13 | +0.87 ± 0.16 |
Full leads in both seeds on all three benchmarks. With two seeds the result is directional, but the MATH500 gap (+1.40) agrees with the 24-seed comparison against LOO (+1.36, Table 40).
Q2. Why learned values outperform the oracle.
In the matched comparison the gap is 1.45 points on MATH500 and is reproducible (one negative seed of 24). Learned credit stays close to the verifier on average but reduces advantage variance by 19–26%, which indicates noise removal. On Code, the arms also differ in the Eq. (4) fallback on low-reward episodes. Our response to Reviewer tjgA (items 1 to 3) gives the details.
Q3. MBPP and MBPP+ scores.
Your diagnosis was correct: the harness omitted the function interface. With the corrected harness, the untrained SFT team's MBPP-test score rises from 5.31%/12.15% to 76.67%/78.07% (n=2/n=3, mean test-pass fraction). With a named entry point and one public example, single-agent strict pass@1 is 72.80% on MBPP-test and 61.90% on MBPP+, in line with published results. MASHAP-LLM leads C3 in all eight corrected cells (Tables 2 and 9). Our response to Reviewer tjgA (Q3) gives the protocol and full results.
Q4. Token and GPU-hour accounting.
The revision separates generated from processed tokens and adds the MATH stage ledger (Table 37) and budget caps (response to Reviewer tjgA, item 4). Fixed-schedule costs and budget endpoints are separated in Weakness 2, and the caption of Table 43 now notes the difference.
Q5. Equal compute in the main tables.
No. The headline result is the GPU-hour envelope (Table 34, originally 31). Full exceeds the strongest competitor, SEPO, by 1.79 points on MATH500 (54.36 vs. 52.57, corrected interval [0.96, 2.63]) and 0.64 on Codeforces (6.50 vs. 5.86, [0.16, 1.12]), with the highest mean against all six baselines on all four benchmarks. The token envelope gives +1.73 and +0.63 (Table 35). The abstract and introduction now report this result.
Follow-up to Reviewer opKq
Thank you again for your review. As the discussion period is closing, we would like to check whether our responses address your concerns. In brief:
- Shaping control. We ran the control you proposed, equal allocation with LLM-SHAP values as LOO shaping. MASHAP-LLM leads in both seeds on all three math benchmarks (+1.40 on MATH500, Table 41), in line with the 24-seed comparison against LOO (+1.36, Table 40).
- MBPP. The harness issue you diagnosed is fixed. The untrained SFT team now scores 76.67%/78.07% (n=2/n=3), and MASHAP-LLM leads C3 in all eight corrected cells.
- Compute. The headline result is now the GPU-hour envelope, where MASHAP-LLM exceeds the strongest of six baselines by 1.79 points on MATH500 and 0.64 on Codeforces. The token tables separate generated from processed tokens, with a stage ledger in Table 37.
Reviewer tjgA has confirmed that the token accounting, MBPP harness and Eq. (4) issues are resolved. We would be grateful if you could consider whether the revision changes your assessment, and we are happy to clarify any remaining point.
Thank you for the constructive review and for proposing the shaping control. In short: (1) with the learned values fixed, Shapley allocation exceeds LOO credit, additive role credit and the equal-allocation shaping control you proposed. (2) The main tables are not compute-matched. Under a common GPU-hour budget, MASHAP-LLM exceeds the strongest of six baselines by 1.79 points on MATH500 and 0.64 on Codeforces, and the abstract now reports this result. (3) The low MBPP scores came from our harness and are corrected.
Weaknesses
1. Whether the gains come from credit assignment.
The most direct test fixes the learned coalition values and varies only how they become rewards. Shapley allocation exceeds LOO by 1.36/0.49 points and Additive-role by 1.67/0.61 on MATH500/Codeforces, with positive corrected intervals (24/12 paired seeds, Table 40, originally 36). The shaping control agrees, with Full leading in both seeds on all three math benchmarks (Q1, Table 41). Task alignment also matters. Full exceeds Task-shuffled, which changes only the task-to-credit assignment, by 2.08/0.63 points.
The gap in Table 3 (54.4 vs. 49.6) also reflects different starting states and seed sets. In the matched comparison (Table 40), learned values exceed verifier values by 1.45/0.53 points (Q2).
Table 19 (originally 18) measures the likelihood of the same recorded final answer after upstream information is removed. A small change means the recorded answer remains plausible without the Reasoner's message, not that the Reasoner is unimportant. When downstream roles regenerate, removing the Reasoner changes the verifier score on 14.2%/18.8% of two-/three-role MATH episodes, 14.2%/14.6% on CMATH and 82.4%/83.8% on Code, where partial test credit makes changes frequent. With two roles these rates equal VD in Table 28, since the coalition without the Reasoner is the only answer-capable strict coalition. On successful episodes the Reasoner receives a mean share of 0.10–0.13 (Table 30).
2. Compute and baseline performance.
Tables 1, 2 and 32 (originally 29) use each method's own fixed schedule and are not compute-matched. For seed 103 (three roles, MATH), Full used 2,299.7 GPU-h, and the reconstructed costs of C3, CCPO and SHARP are 571.9, 568.6 and 570.7 GPU-h. Fixed-schedule and envelope runs used different cluster conditions, so per-update costs differ (Full's 8,000 fixed updates cost about 640 GPU-h less than envelope rates predict) and cross-protocol ratios are approximate. The closest-cost comparison is Equal, which shares our PPO scaffold and update count at 0.80× our GPU-hours (1,849.2 GPU-h). Full leads it by 8.66 points on MATH500 (54.23 vs. 45.57, 24 paired seeds).
Under the GPU-hour envelope, C3 runs 617 updates instead of 148, about four times as many, and improves from 48.19 to 52.34. MASHAP-LLM completes 6,481 of its 8,000 updates with an essentially unchanged score (54.36 vs. 54.23), indicating a plateau. At 25/50/75/100% of the GPU allowance, Full's margin over the better of C3 and SEPO is −4.42, +0.74, +1.20 and +1.79 on MATH500 and −1.71, +0.33, +0.49 and +0.64 on Codeforces. The early deficit reflects the fitting phase, which costs 654.7 GPU-h on MATH, 27.3% of the 2,400 GPU-h cap (Table 37). The resource-matched headline is given under Q5, and the MBPP correction under Q3.