- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MASContributionBench, a benchmark covering seven multi-agent system (MAS) architectures, seven tasks, three attribution methods (leave-one-out, Shapley, and Banzhaf), and three interventions (topology, role prompts, and runtime permissions). With results from MASContributionBench, the authors make the following observations.
- A random team (MAS) can perform similarly as a pre-defined MAS configuration, and different topologies have similar performance on SWE-Bench (verified). These results suggest that aggregate performance does not provide evidence that one MAS architecture is better than the other.
- Leave-one-out (LOO) attribution has different score distributions compared to Shapley and Banzhaf attributions, and scores are context-dependent.
- Across interventions based on topology, role prompts, and runtime permissions, results show that contribution can change even for the non-intervened agents, suggesting the fluidity of how different agents are intended to contribute.
Critical strengths
- The benchmark covers a wide range of tasks, MAS topologies, attribution methods, and intervention methods. I hope the authors can release the traces and performance data points for future work to build upon.
- The results in Section 3.4 are insightful in my opinion. Across interventions based on topology, role prompts, and runtime permissions, results show that contribution can change even for the non-intervened agents, suggesting the fluidity of how different agents are intended to contribute. In other words, these results show that we cannot reason which agents are important simply based on the null topology, roles, and permissions. I think this framing is worth highlighting more in the paper.
Critical weaknesses
- Although this paper provides insights, these insights are not demonstrated in downstream use cases such as MAS optimization (e.g. by removing unnecessary agents or removing unnecessary communication edges).
- The differences between LOO and coalition-based attributions such as Shapley and Banzhaf have already be observed in prior work [1].
References [1] Lu et al. (2026) Agents that matter: Optimizing multi-agent LLMs via removal-based attribution.
Questions for authors
- Do the insights observed from your results actually lead to a better optimized MAS configurations?
- The percentages of resolved tasks in SWE-Bench (Verified) are low across the topologies. Could this be due to the low capability of Qwen3.8-27B-FP8? In other words, it is possible that the similarity between different topologies does not hold for an LLM more capable at coding.
- Do the insights observed from your paper hold using a different LLM?
8 comments
Response 1/3
- Single-agent pruning reveals potential performance–cost trade-offs that require further validation.
To examine whether our findings can inform team configuration, we conducted a preliminary pruning study on the A3 DAG. We compared the complete team with five modified teams, each obtained by removing one eligible intermediate agent: Planner, Retriever, Coder, Debugger, or Tester. For each removal, we connected the agent's predecessors to its successors and retained the Finalizer.
This study evaluates single-agent pruning candidates only. It does not evaluate joint removal of multiple roles or a fully redesigned team configuration. All conditions were evaluated on the same discovery tasks and seeds: 30 GPQA Diamond tasks and 30 IFBench tasks, with three seeds per task.
Table 1 reports utility and average total token use. Utility is normalized exact-match accuracy on GPQA and strict prompt-level instruction-following pass rate on IFBench. For each displayed candidate, the random-removal comparator is the uniform average over the other four eligible single-agent removals.
Table 1: Preliminary single-agent pruning results on discovery tasks. is the utility difference relative to the complete team, in percentage points (pp). For each displayed candidate, the paired 95% confidence interval is computed by task-cluster bootstrap, retaining all three seeds and compared conditions for each sampled task. Tokens are average input and output tokens per run.
| Benchmark | Team configuration | Utility | 95% CI for | Tokens | |
|---|---|---|---|---|---|
| GPQA | Complete team | 50.00% | — | — | 19,165 |
| GPQA | Remove Debugger | 61.11% | pp | pp | 13,625 |
| GPQA | Random: other four | 55.28% | pp | — | 16,728 |
| IFBench | Complete team | 89.66% | — | — | 7,747 |
| IFBench | Remove Coder | 89.66% | pp | pp | 6,654 |
| IFBench | Random: other four | 85.92% | pp | — | 6,252 |
On GPQA, removing the Debugger increased observed utility by 11.11 percentage points and reduced average token use by 28.9% relative to the complete team. Its observed utility was 5.83 points above the average for removing one of the other four roles (paired 95% CI: points). On IFBench, removing the Coder produced the same observed utility as the complete team while reducing average token use by 14.1%. Its utility was 3.74 points above the other-four random-removal average, but the paired 95% CI was points. Moreover, its token use was higher than that random-removal average. Thus, the IFBench evidence does not establish an advantage over random removal.
These random comparisons concern specific removal candidates selected after inspecting the discovery results. They do not constitute an end-to-end comparison between an attribution-guided selection strategy and a random selection strategy. Indeed, the GPQA results expose a concrete limitation of the original null-replacement attribution. Defining LOO as complete-team utility minus intervened-team utility, the Debugger's null-replacement LOO was points, whereas its real-removal LOO was points. Ranking roles by the lowest null-replacement LOO therefore did not place the Debugger among the leading removal candidates, even though its real removal had the highest observed utility. Null replacement retains the agent's graph position, while real removal also changes communication paths; the former should not be treated as a direct prediction of the latter.
The GPQA candidate also requires a budget-controlled follow-up. The complete team's Finalizer reached its 4,096-token output limit in 38 of 90 runs, compared with 29 of 90 after Debugger removal. All ten paired runs that improved after removal had reached the limit in the complete-team condition. Among the 51 paired runs, spanning 19 tasks, in which neither condition reached the limit, both conditions passed 45 runs. This post-hoc subset comparison is descriptive: conditioning on outcomes observed after both interventions can introduce selection bias and does not isolate the causal effect of the output budget. The GPQA utility advantage should therefore not yet be treated as a validated improvement.
Both candidates in Table 1 were identified after inspecting the discovery results and have not been tested on independent tasks. The present study therefore provides testable pruning candidates and evidence about the limits of null-replacement attribution, but it does not yet establish an attribution-guided configuration improvement on new tasks.
Response 2/3
- SWE-bench Verified: backbone capacity and architectural similarity.
To examine whether the low completion rates and apparent similarity across architectures are specific to Qwen3.8-27B-FP8, we evaluated three additional backbones under our experimental scaffold. We first tested Gemma-4-31B-it-QAT, a different open-weight model, and observed similarly low completion rates. After analyzing the failure cases, we evaluated Qwen3-Coder-30B-A3B-Instruct-FP8, a coding-oriented model represented on the SWE-bench leaderboard. We further evaluated Claude Opus 4.5 as a stronger closed-weight backbone.
For each backbone, we evaluated A2 Chain and A3 DAG on the same 30 task instances, with three runs per instance. The table reports confirmed resolved runs out of 90 planned runs per condition.
| Backbone | Architecture | Confirmed resolved / Planned | Rate |
|---|---|---|---|
| Qwen3.8-27B-FP8 | A2 Chain | 5 / 90 | 5.6% |
| Qwen3.8-27B-FP8 | A3 DAG | 4 / 90 | 4.4% |
| Gemma-4-31B-it-QAT | A2 Chain | 3 / 90 | 3.3% |
| Gemma-4-31B-it-QAT | A3 DAG | 6 / 90 | 6.7% |
| Qwen3-Coder | A2 Chain | 3 / 90 | 3.3% |
| Qwen3-Coder | A3 DAG | 3 / 90 | 3.3% |
| Claude Opus 4.5 | A2 Chain | 17 / 90 | 18.9% |
| Claude Opus 4.5 | A3 DAG | 14 / 90 | 15.6% |
Each condition comprises 30 distinct tasks and three runs per task. Rates use all 90 planned runs as the denominator and report confirmed resolved runs.
The results suggest that low completion is not unique to Qwen3.8-27B-FP8 under our scaffold: both Gemma and the coding-oriented Qwen3-Coder also produced few confirmed solutions. Claude achieved more confirmed resolutions than the other evaluated backbones in both architectures, indicating that backbone choice matters in this setting. Nevertheless, its confirmed completion rates remained below 20%.
For Claude, Chain and DAG resolved 17 and 14 of the 90 planned runs, respectively, a difference of 3.3 percentage points. Thus, we still observe a small absolute gap between these two architectures with this closed-weight backbone. However, this observation does not establish architectural equivalence: the sample is limited, some outcomes remain unknown, and A2 and A3 differ in their role sets as well as their communication structures.
Our failure analysis shows that during generation, agents received only the issue description and could not browse the repository, edit files, or execute tests. A substantial fraction of submitted patches could not be applied to the target repository; thus, completion rates reflect pipeline constraints as well as backbone behavior, and should not be read as standalone coding ability.
These findings address the concern in a bounded setting: replacing Qwen3.8 with other backbones, including a coding-oriented model and Claude Opus 4.5, did not eliminate low completion rates, and the absolute Chain–DAG gap remained small for Claude. We cannot rule out that a stronger model combined with repository access and iterative test feedback would reveal differences that are difficult to observe here. We therefore do not generalize the observed similarity to tool-enabled coding agents or attribute it solely to topology.
SWE-bench Verified nevertheless provides a useful repository-level evaluation: it tests whether outputs from a multi-agent workflow can become patches that apply to real repositories and pass their tests. In our setting, it exposes a gap between textual collaboration and an executable software artifact. We interpret this finding as a limitation of the current execution setting and as motivation for future repository-aware, tool-enabled comparisons, rather than as a definitive ranking of architectures.
Summary Response
Dear Reviewer tsr5,
We thank you for recognizing the breadth of our benchmark and the insights from our intervention results. We also appreciate your suggestion to highlight how the measured contribution of an unchanged agent can vary with the surrounding system.
Distinction from Agents that Matter. We agree that differences between LOO and coalition-based attributions have been observed in prior work and do not claim this general observation as novel. Our work complements it through controlled interventions and execution audits, examining how routing, role prompts, and permissions affect measured contributions and how configured interventions are realized during execution. (
Weakness 2)We summarize our responses to your questions as follows:
We conduct a preliminary single-agent pruning study to examine potential performance–cost trade-offs. The results identify candidates for further testing and illustrate why null-replacement attribution does not directly predict removability. The candidates have not yet been validated on new tasks. (
Weakness 1; Response 1/3)We evaluate three additional backbones on SWE-bench Verified, including a coding-oriented model and Claude Opus 4.5. We observe low confirmed completion rates under our scaffold and clarify the limitations of comparing architectures in this setting. (Response 2/3)
We examine the Verifier topology effect in homogeneous teams with additional backbones and a heterogeneous model team. The results support context-dependent attribution in the tested settings, while the observed sign reversal and its magnitude are not universal. (Response 3/3)
We will clarify the scope of our findings and highlight the interpretation of our intervention results in the paper. We hope our responses address your concerns and look forward to further discussion with you.