- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces a framework for situating different experimental setups for understanding attribution in multi-agent systems (MAS). They evaluate a collection of different evaluation protocols (which they call "interventions") and highlight different patterns that arise depending on the experimental setup.
Critical strengths
- The topic is important to study. Clearly the world is shifting towards multi-agent systems (MAS) and the problem of agent attribution has many important downstream consequences.
- The framework introduced seems quite extensive and covers many of the SOTA methods.
Critical weaknesses
- It is unclear to me what the technical novelty of this work is. I understand that it was a very significant engineering task and I really appreciate the effort that this entails so I am not trying to minimize the work involved. However, I feel that the work is much more suited to a benchmark track than a main research track.
- There is very little generalizable guidance on how these findings should be used to extend to other datasets or future work. After gathering all the "Key Takeaways" -- (1) Agent importance is not a single fixed quantity, (2) LOO can serve as an effective proxy for combinatorial attribution, (3) Attribution guided interventions offer the strongest performance-cost trade offs..., (4) Matching attribution to the intervention improves target selection, agent importance depends on evaluation objective --> all of these highlight broad patterns that arise in your work but they are not tied to concrete next steps. There is no guidance on what makes a specific task more or less challenging. Which intervention techniques are more or less insightful in a particular setting. Furthermore, it seems that your comparisons highlight that specific intervention techniques are more or less effective (e.g. introspective removal seems to overestimate coalition utility relative to agent ablation). It is unclear why that is and what this means for future practitioners.
Questions for authors
- At the end of the experimental section I was still unsure... are these attribution methods actually identifying the agents whose intervention produces the greatest improvement, or are they simply agreeing with one another? Can you please clarify.
- Can you speak to differences between the utility of these various experimental setups with respect to the specific tasks you explored in your various datasets?
8 comments
Re: Generalization of the above guidelines to new agent systems (W2)
We further demonstrate the applicability of our guidelines to two real-world MAS applications: 1) MDTeamGPT [4] for medical decision-making and 2) MetaGPT [5] for software engineering. MDTeamGPT features a more complex architecture, while MetaGPT follows a sequential workflow. Note that these two agentic systems extend beyond the topologies we evaluated in Sec. 4.2
-
Objective-specific, cost-efficient MAS deployment. In our original submission, we aimed to balance performance and inference costs for MDTeamGPT (Sec. 5.5), which employs closed-source GPT-5 mini. We therefore consider targeted model replacement (GPT-5-mini → Qwen 27B), as replacing all agents can decrease closed-source token usage but degrade performance.
Using LOO attribution under two objectives—accuracy and ethical safety- we identify agents with minimal or negative contributions to ethical safety, and that degrade accuracy minimally (Specialist, Triage, and CoT Reviewer). Targeted replacement of these agents improves ethical performance by 10% (table below), while reducing closed-source model token usage by 78.8% on MedEthicsQA (below) and 53.4% on MedQA (Fig. 24).
Table 5: MDTeamGPT on MedEthicsQA (mean acc. (%) / closed-source tokens (M))
Protocol k = 0 k = 1 k = 2 k = 3 Full team 73.3 / 4.3 — — — Model replacement — 78.9 / 1.5 78.8 / 1.1 78.9 / 0.9 Agent ablation — 73.9 / 3.3 74.0 / 3.3 69.9 / 3.0 Agent ablation (by token usage) — 73.1 / 4.2 65.8 / 3.9 57.9 / 3.7 -
Generalizability of prompt LOO to a new MAS. Building on those guidelines, we proposed Prompt LOO and evaluated it on MetaGPT, a sequential software-engineering MAS, using the MBPP code-generation benchmark (n=100) with Gemini 2.5 Flash as the backbone. We follow the same prompt optimization protocol as in our main experiments and compare prompt LOO against alternative intervention strategies.
As shown below, prompt LOO improves pass@1 from 62.3% to 85.7% at (k=3), outperforming model replacement (79.0%) and agent ablation (55.3%). In contrast, token-usage-guided ablation reduces performance to 31.7%. These results demonstrate that prompt LOO remains effective in a different MAS architecture and task setting, supporting the generalizability of our guidelines on attribution-based, targeted optimization.
Table 6: MetaGPT on MBPP (pass@1, 95% CI).
Method k=1 k=2 k=3 Full team 62.3 (2.7) 62.3 (2.7) 62.3 (2.7) Model replacement 72.0 (4.0) 71.0 (3.0) 79.0 (2.0) Agent ablation 65.3 (1.9) 62.0 (1.6) 55.3 (2.5) Agent ablation by token usage 63.7 (2.6) 56.7 (2.4) 31.7 (5.4) Prompt optimization 76.3 (0.9) 83.0 (1.4) 85.7 (2.5)
Together, these case studies provide further evidence that our guidelines generalize across different MAS architectures and task distributions. We’ve included results in the revised manuscript (Appendices F and G).
Re: Novelty and contributions (W1)
We thank the reviewer for raising this concern. Prior work [6–8] on agent attribution has largely examined fixed combinations of attribution design choices, often implicitly treating the resulting scores as a universal measure of agent importance. However, these design choices have not been explicitly formalized, nor have their implications been systematically examined in the context of MAS attribution. To our knowledge, we are the first to connect and formalize these design choices within a unified framework, making their underlying assumptions explicit and enabling systematic examination of how they affect agent importance. Building on these insights, we further introduce prompt LOO (Sec. 5.4), an efficient attribution approach that measures agent-level contributions under prompt optimization interventions. Together, we believe these contributions go beyond empirical benchmarking by providing a systematic foundation for agent attribution and guiding the development of new methods for MAS. We also revised the introduction to make our contribution clearer.
Re: Whether attribution methods identify effective intervention targets and whether attribution rankings agree (Q1)
We thank the reviewer for this important question. We clarify two related but distinct aspects:
-
Do attribution scores identify targets for effective interventions?
We directly assess this through executed top- intervention experiments (Figures 20–22), where we intervene on agents selected by attribution rankings and measure the resulting changes in system performance. As controls, we include random ranking and ranking based on agent token usage in the full system to assess whether attribution-guided selection outperforms these simple alternatives. The table below reports intervention AUCs on PlanCraft, averaged across topologies.
Intervention LOO Shapley Random Token usage Agent ablation 0.298 (0.01) 0.295 (0.01) 0.234 (0.01) 0.268 (0.01) Introspective removal 0.419 (0.013) 0.426 (0.014) 0.392 (0.013) 0.357 (0.014) Model replacement 0.379 (0.02) 0.392 (0.02) 0.315 (0.02) 0.317 (0.01) Prompt optimization 0.418 (0.02) 0.423 (0.02) 0.366 (0.02) 0.365 (0.01) As shown above, both LOO and Shapley consistently outperform random and token-usage rankings across the intervention protocols. These results provide direct evidence that attribution methods identify effective intervention targets and improve system performance.
-
Do attribution scores computed under different distributions yield similar agent rankings?
We observe that combinatorial attribution methods, e.g., Shapley, Myerson, Owen values, generally produce highly consistent agent rankings despite differences in their coalition weightings (Figure 10). This agreement can arise from mathematical equivalences between kernels under certain topologies (e.g., Owen reducing to Shapley under a single worker group, and Myerson under a complete attribution graph), or from structurally indispensable agents (e.g., the Coder or orchestrator) that consistently receive high attribution across methods.
However, ranking differences become more pronounced in settings with explicit group structures, such as hybrid and centralized topologies, where different kernels account for structural dependencies differently. Moreover, while LOO often identifies the same most influential agents as combinatorial methods, their rankings of less influential agents differ, with an average Spearman rank correlation of approximately 0.73 (Figure 10). Thus, attribution methods exhibit substantial agreement on key agents but do not necessarily produce identical rankings.
In summary, attribution methods do identify effective intervention targets (agents) that improve system performance, while ranking agreement depends on factors such as MAS topology and task-specific agent capabilities.
Dear Reviewer CaCQ,
We thank the reviewer for these thoughtful comments and for recognizing our efforts. We address the concerns below and welcome further questions.
Re: Implications of our findings for practitioners (W2)
Our findings provide actionable guidance for agent attribution in MAS along four dimensions: attribution kernels, intervention protocol, computational budget, and evaluation objective.
Attribution kernels: Our results show that LOO is effective and substantially cheaper than Shapley across settings, including 10-agent systems where it outperforms budget-matched sampled Shapley (see our response to h69T, W1). Thus, practitioners can start with LOO to identify influential agents.
Intervention protocols:
Agent ablation is useful in modular systems (e.g., independent workflows), especially for identifying essential or redundant agents. However, in tightly coupled workflows, agent ablation may disrupt downstream interactions and obscure other agents' contributions. In such settings, prompt optimization and model replacement offer structure-preserving alternatives that can yield substantial performance gains, particularly in centralized and hybrid topologies (see the table below for mean intervention gains across datasets, in percentage points), albeit at higher token costs (Fig. 6).
When is introspective removal useful, and under what conditions is it reliable? (Sec. 5.1) Our results show that LLM-estimated coalition utilities often diverge from actual ablation outcomes, particularly when complex agent interactions are involved. However, introspective removal may still be useful for identifying structurally indispensable agents (e.g., an Orchestrator whose removal severely disrupts system performance). Given its low attribution cost, it can serve as a preliminary screening tool under tight token budgets, although its reliability depends on the system configuration.
Practitioners should therefore choose based on agent structure, intended system modification, and computational budget, as summarized below.
Intervention protocols: (1) introspective removal; (2) agent ablation; (3) model replacement; (4) prompt optimization.
Prioritize high-impact agents with Prompt LOO (Sec. 5.4): As token costs become increasingly important in MAS [1], we introduce prompt LOO to guide selective deployment of optimized agent prompts rather than modifying the entire system. Our top-k experiments (Fig. 6; Table 3 below) demonstrate that attribution-guided selection achieves favorable performance–cost trade-offs. Moreover, it outperforms broader system-level optimization methods, including Meta-Harness [2] (context and tool management) and AFlow [3] (workflow orchestration), in our evaluated settings (Table 4 below). These findings suggest that practitioners can leverage Prompt LOO to identify and prioritize influential agents for targeted prompt optimization and deployment, reducing inference cost.
Table 3: Mean success rate (%) and inference token usage (M) averaged across topologies
Table 4: PlanCraft success rate (%): Top-1 / Top-2 optimized
Agent attribution can be multidimensional (Sec. 5.5). Our formulation explicitly shows that attribution depends on the evaluation objective, while our medical auditing case study demonstrates that agent rankings can vary across objectives in practice. Thus, practitioners should avoid assuming a universal importance ranking and consider how interventions targeting one objective may affect other system behaviors.
Together, we believe these findings provide actionable guidelines for practitioners applying agent attribution in MAS. We've also revised the introduction, key takeaways, and results sections to make these implications more explicit.