acceptodds
Under review as a conference paper at ICLR 2027

Agents that Matter: How Subtle Choices Shape Agent Attribution in Multi-Agent Systems

Abstract

As multi-agent systems (MAS) become increasingly complex, identifying the contributions of agents is critical for system optimization. However, existing approaches lack a unified formulation for credit assignment. In this work, we formalize agent attribution as a cooperative game, parameterized by the coalition distribution, intervention protocol, and target metric. Using this framework, we demonstrate that intervention protocols induce distinct games: Agent ablation isolates structural bottlenecks, whereas introspective LLM judges fail to faithfully approximate this behavior. We further find that Leave-One-Out (LOO) identifies bottleneck agents as effectively as combinatorial methods at much lower computational cost. Treating prompt optimization as an intervention protocol, we show that prompt-conditioned attribution can identify optimization targets and improve MAS performance. Finally, we apply our framework to audit a medical MAS, revealing that agent contributions to diagnostic accuracy and ethical behavior are often decoupled. By intervening on counterproductive roles, we observe an increase in ethics alignment while maintaining diagnostic accuracy. Overall, this work provides a principled approach for cost-effective MAS attribution and intervention. Code for reproducing our experiments is available at https://anonymous.4open.science/r/agent-eval-B850

Primary area:
interpretability and explainable AI
Keywords:
Multi-agent systems, LLM agents, Agent attribution, Credit assignment, Cooperative game theory, Interpretability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CaCQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a framework for situating different experimental setups for understanding attribution in multi-agent systems (MAS). They evaluate a collection of different evaluation protocols (which they call "interventions") and highlight different patterns that arise depending on the experimental setup.

Critical strengths
  1. The topic is important to study. Clearly the world is shifting towards multi-agent systems (MAS) and the problem of agent attribution has many important downstream consequences.
  2. The framework introduced seems quite extensive and covers many of the SOTA methods.
Critical weaknesses
  1. It is unclear to me what the technical novelty of this work is. I understand that it was a very significant engineering task and I really appreciate the effort that this entails so I am not trying to minimize the work involved. However, I feel that the work is much more suited to a benchmark track than a main research track.
  2. There is very little generalizable guidance on how these findings should be used to extend to other datasets or future work. After gathering all the "Key Takeaways" -- (1) Agent importance is not a single fixed quantity, (2) LOO can serve as an effective proxy for combinatorial attribution, (3) Attribution guided interventions offer the strongest performance-cost trade offs..., (4) Matching attribution to the intervention improves target selection, agent importance depends on evaluation objective --> all of these highlight broad patterns that arise in your work but they are not tied to concrete next steps. There is no guidance on what makes a specific task more or less challenging. Which intervention techniques are more or less insightful in a particular setting. Furthermore, it seems that your comparisons highlight that specific intervention techniques are more or less effective (e.g. introspective removal seems to overestimate coalition utility relative to agent ablation). It is unclear why that is and what this means for future practitioners.
Questions for authors
  1. At the end of the experimental section I was still unsure... are these attribution methods actually identifying the agents whose intervention produces the greatest improvement, or are they simply agreeing with one another? Can you please clarify.
  2. Can you speak to differences between the utility of these various experimental setups with respect to the specific tasks you explored in your various datasets?
8 comments

Dear Reviewer CaCQ,

We thank the reviewer for these thoughtful comments and for recognizing our efforts. We address the concerns below and welcome further questions.

Re: Implications of our findings for practitioners (W2)

Our findings provide actionable guidance for agent attribution in MAS along four dimensions: attribution kernels, intervention protocol, computational budget, and evaluation objective.

  1. Attribution kernels: Our results show that LOO is effective and substantially cheaper than Shapley across settings, including 10-agent systems where it outperforms budget-matched sampled Shapley (see our response to h69T, W1). Thus, practitioners can start with LOO to identify influential agents.

  2. Intervention protocols:

    • Agent ablation is useful in modular systems (e.g., independent workflows), especially for identifying essential or redundant agents. However, in tightly coupled workflows, agent ablation may disrupt downstream interactions and obscure other agents' contributions. In such settings, prompt optimization and model replacement offer structure-preserving alternatives that can yield substantial performance gains, particularly in centralized and hybrid topologies (see the table below for mean intervention gains across datasets, in percentage points), albeit at higher token costs (Fig. 6).

      InterventionInd.Dec.Cent.Hybrid
      Agent ablation+3.0+1.0+0.8+1.5
      Model replacement+4.5+2.0+5.7+9.5
      Prompt optimization+5.5+8.5+9.5+9.5
    • When is introspective removal useful, and under what conditions is it reliable? (Sec. 5.1) Our results show that LLM-estimated coalition utilities often diverge from actual ablation outcomes, particularly when complex agent interactions are involved. However, introspective removal may still be useful for identifying structurally indispensable agents (e.g., an Orchestrator whose removal severely disrupts system performance). Given its low attribution cost, it can serve as a preliminary screening tool under tight token budgets, although its reliability depends on the system configuration.

    • Practitioners should therefore choose based on agent structure, intended system modification, and computational budget, as summarized below.

      Compute availableInd.Dec.Cent.Hybrid
      Low — O(11)(1)+LOO(1)+LOO(1)+LOO(1)+LOO
      Medium — O(nn)(2)(4)+LOO(3)(4)+LOO(3)(4)+LOO(3)(4)+LOO
      High — O(2n2^n)(2)(4)+Shapley(3)(4)+Shapley/Myerson(3)(4)+Shapley/Owen(3)(4)+Shapley/Owen

      Intervention protocols: (1) introspective removal; (2) agent ablation; (3) model replacement; (4) prompt optimization.

  3. Prioritize high-impact agents with Prompt LOO (Sec. 5.4): As token costs become increasingly important in MAS [1], we introduce prompt LOO to guide selective deployment of optimized agent prompts rather than modifying the entire system. Our top-kk experiments (Fig. 6; Table 3 below) demonstrate that attribution-guided selection achieves favorable performance–cost trade-offs. Moreover, it outperforms broader system-level optimization methods, including Meta-Harness [2] (context and tool management) and AFlow [3] (workflow orchestration), in our evaluated settings (Table 4 below). These findings suggest that practitioners can leverage Prompt LOO to identify and prioritize influential agents for targeted prompt optimization and deployment, reducing inference cost.

    Table 3: Mean success rate (%) and inference token usage (M) averaged across topologies

    DatasetAll prompts optimizedTop-2 optimized
    PlanCraft42.0(5.44M)42.0(2.45M)
    WorkBench39.0(1.78M)39.0(0.74M)
    BrowseComp-Plus51.3(8.55M)50.7(3.35M)
    LCB66.0(3.40M)67.5(1.51M)

    Table 4: PlanCraft success rate (%): Top-1 / Top-2 optimized

    Method
    Base39.3(1.00)
    CG-OPO40.0(1.01)/40.0(1.01)
    Prompt LOO41.3(1.0)/42.0(1.01)
    Random40.6(0.93)/41.4(0.96)
    Token usage39.3(1.00)/41.3(1.01)
    --
    MetaHarness0.395(0.06)
    AFLow0.385(0.06)
  4. Agent attribution can be multidimensional (Sec. 5.5). Our formulation explicitly shows that attribution depends on the evaluation objective, while our medical auditing case study demonstrates that agent rankings can vary across objectives in practice. Thus, practitioners should avoid assuming a universal importance ranking and consider how interventions targeting one objective may affect other system behaviors.

Together, we believe these findings provide actionable guidelines for practitioners applying agent attribution in MAS. We've also revised the introduction, key takeaways, and results sections to make these implications more explicit.

Re: Generalization of the above guidelines to new agent systems (W2)

We further demonstrate the applicability of our guidelines to two real-world MAS applications: 1) MDTeamGPT [4] for medical decision-making and 2) MetaGPT [5] for software engineering. MDTeamGPT features a more complex architecture, while MetaGPT follows a sequential workflow. Note that these two agentic systems extend beyond the topologies we evaluated in Sec. 4.2

  • Objective-specific, cost-efficient MAS deployment. In our original submission, we aimed to balance performance and inference costs for MDTeamGPT (Sec. 5.5), which employs closed-source GPT-5 mini. We therefore consider targeted model replacement (GPT-5-mini → Qwen 27B), as replacing all agents can decrease closed-source token usage but degrade performance.

    Using LOO attribution under two objectives—accuracy and ethical safety- we identify agents with minimal or negative contributions to ethical safety, and that degrade accuracy minimally (Specialist, Triage, and CoT Reviewer). Targeted replacement of these agents improves ethical performance by 10% (table below), while reducing closed-source model token usage by 78.8% on MedEthicsQA (below) and 53.4% on MedQA (Fig. 24).

    Table 5: MDTeamGPT on MedEthicsQA (mean acc. (%) / closed-source tokens (M))

    Protocolk = 0k = 1k = 2k = 3
    Full team73.3 / 4.3———
    Model replacement—78.9 / 1.578.8 / 1.178.9 / 0.9
    Agent ablation—73.9 / 3.374.0 / 3.369.9 / 3.0
    Agent ablation (by token usage)—73.1 / 4.265.8 / 3.957.9 / 3.7
  • Generalizability of prompt LOO to a new MAS. Building on those guidelines, we proposed Prompt LOO and evaluated it on MetaGPT, a sequential software-engineering MAS, using the MBPP code-generation benchmark (n=100) with Gemini 2.5 Flash as the backbone. We follow the same prompt optimization protocol as in our main experiments and compare prompt LOO against alternative intervention strategies.

    As shown below, prompt LOO improves pass@1 from 62.3% to 85.7% at (k=3), outperforming model replacement (79.0%) and agent ablation (55.3%). In contrast, token-usage-guided ablation reduces performance to 31.7%. These results demonstrate that prompt LOO remains effective in a different MAS architecture and task setting, supporting the generalizability of our guidelines on attribution-based, targeted optimization.

    Table 6: MetaGPT on MBPP (pass@1, 95% CI).

    Methodk=1k=2k=3
    Full team62.3 (2.7)62.3 (2.7)62.3 (2.7)
    Model replacement72.0 (4.0)71.0 (3.0)79.0 (2.0)
    Agent ablation65.3 (1.9)62.0 (1.6)55.3 (2.5)
    Agent ablation by token usage63.7 (2.6)56.7 (2.4)31.7 (5.4)
    Prompt optimization76.3 (0.9)83.0 (1.4)85.7 (2.5)

Together, these case studies provide further evidence that our guidelines generalize across different MAS architectures and task distributions. We’ve included results in the revised manuscript (Appendices F and G).

Re: Novelty and contributions (W1)

We thank the reviewer for raising this concern. Prior work [6–8] on agent attribution has largely examined fixed combinations of attribution design choices, often implicitly treating the resulting scores as a universal measure of agent importance. However, these design choices have not been explicitly formalized, nor have their implications been systematically examined in the context of MAS attribution. To our knowledge, we are the first to connect and formalize these design choices within a unified framework, making their underlying assumptions explicit and enabling systematic examination of how they affect agent importance. Building on these insights, we further introduce prompt LOO (Sec. 5.4), an efficient attribution approach that measures agent-level contributions under prompt optimization interventions. Together, we believe these contributions go beyond empirical benchmarking by providing a systematic foundation for agent attribution and guiding the development of new methods for MAS. We also revised the introduction to make our contribution clearer.

Re: Whether attribution methods identify effective intervention targets and whether attribution rankings agree (Q1)

We thank the reviewer for this important question. We clarify two related but distinct aspects:

  • Do attribution scores identify targets for effective interventions?

    We directly assess this through executed top-kk intervention experiments (Figures 20–22), where we intervene on agents selected by attribution rankings and measure the resulting changes in system performance. As controls, we include random ranking and ranking based on agent token usage in the full system to assess whether attribution-guided selection outperforms these simple alternatives. The table below reports intervention AUCs on PlanCraft, averaged across topologies.

    InterventionLOOShapleyRandomToken usage
    Agent ablation0.298 (0.01)0.295 (0.01)0.234 (0.01)0.268 (0.01)
    Introspective removal0.419 (0.013)0.426 (0.014)0.392 (0.013)0.357 (0.014)
    Model replacement0.379 (0.02)0.392 (0.02)0.315 (0.02)0.317 (0.01)
    Prompt optimization0.418 (0.02)0.423 (0.02)0.366 (0.02)0.365 (0.01)

    As shown above, both LOO and Shapley consistently outperform random and token-usage rankings across the intervention protocols. These results provide direct evidence that attribution methods identify effective intervention targets and improve system performance.

  • Do attribution scores computed under different distributions yield similar agent rankings?

    We observe that combinatorial attribution methods, e.g., Shapley, Myerson, Owen values, generally produce highly consistent agent rankings despite differences in their coalition weightings (Figure 10). This agreement can arise from mathematical equivalences between kernels under certain topologies (e.g., Owen reducing to Shapley under a single worker group, and Myerson under a complete attribution graph), or from structurally indispensable agents (e.g., the Coder or orchestrator) that consistently receive high attribution across methods.

    However, ranking differences become more pronounced in settings with explicit group structures, such as hybrid and centralized topologies, where different kernels account for structural dependencies differently. Moreover, while LOO often identifies the same most influential agents as combinatorial methods, their rankings of less influential agents differ, with an average Spearman rank correlation of approximately 0.73 (Figure 10). Thus, attribution methods exhibit substantial agreement on key agents but do not necessarily produce identical rankings.

In summary, attribution methods do identify effective intervention targets (agents) that improve system performance, while ranking agreement depends on factors such as MAS topology and task-specific agent capabilities.

Reviewer_ZDigOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the problem of Multi-Agent System (MAS), how can one determine which agent is the most important, and which agent should be prioritized for optimization?

Critical strengths

interesting problem

Critical weaknesses

1.The paper claims that using "attribution" to identify which agents should be prioritized for optimization helps reduce overall optimization costs. However, Section 4.4 explicitly states that the authors first use GEPA to jointly optimize all agent prompts, and only then calculate attribution based on these already-optimized prompts to select the top-k agents. so, to determine which agents to optimize, the authors have, in effect, already optimized all of them. The core issue is this: if one must optimize all agents before deciding which ones to optimize, what is the actual advantage of "attribution-guided optimization"?

2.The core formula presented in Section 3 is, in essence, a unified expression of existing concepts from Cooperative Game Theory and Shapley Attribution. The three dimensions mentioned are all design choices that already exist within the field of attribution.

  1. Section 5.4 compares some baselines, it is insufficient to demonstrate the necessity of "Attribution-guided Optimization" itself.

  2. Moreover, I’m curious: in many cases, isn’t the interaction itself the actual problem? In multi-agent systems (MAS), does it even make sense to determine which agent is the "most important" or which one should be prioritized for optimization? Often, such a question is meaningless,take, for instance, research on emergent social risks or risks arising from agent interactions.

From many paper i read, In Multi-Agent Systems (MAS), many failures do not stem from problems within an individual agent itself, but rather from issues that arise during the interactions between multiple agents. While Shapley does account for interaction effects between agents, it still assigns these effects to individual agents.

Questions for authors

see weakness

9 comments

Dear Reviewer ZDig

We thank the reviewer for their thoughtful feedback and insightful questions. Below, we address each concern and welcome further discussion.

Re W1: Benefits of attribution-guided prompt optimization

We thank the reviewer for raising this important point. We clarify that, for prompt optimization, our setup aims to select which optimized agent prompts to deploy. A key motivation is that deploying jointly optimized prompts for all agents does not consistently improve MAS performance, with gains varying across tasks and topologies [1]. Moreover, prior work shows that agents contribute unequally to system performance [2], with some potentially harming collective outcomes [3]. These findings motivate measuring the marginal effect of deploying each agent's optimized prompt to guide selective deployment. We have clarified the definition of this intervention in the revised manuscript (Lines 180, 247) and ensured consistent terminology throughout.

Beyond improving MAS performance, selective deployment offers the additional benefit of reducing recurring inference costs [4]. While the upfront prompt optimization cost remains unchanged, deploying all optimized prompts can incur substantial additional token usage. Here, we quantify the success rates and inference token usage shown in Figure 6 with the table below. Deploying optimized prompts for only the two highest-attribution agents achieves comparable or better performance with 2.2–2.6× fewer additional inference tokens. These findings highlight the practical value of attribution-guided selective deployment in balancing performance and inference costs.

Table 1: Mean success rate (%) and inference token usage (M) averaged across topologies

DatasetAll prompts optimizedTop-2 optimized
PlanCraft42.0 (5.44M)42.0 (2.45M)
WorkBench39.0 (1.78M)39.0 (0.74M)
BrowseComp-Plus51.3 (8.55M)50.7 (3.35M)
LCB66.0 (3.40M)67.5 (1.51M)

Re W2: Contribution of the unified agent attribution framework (Sec. 3).

We thank the reviewer for raising this concern. While these concepts are well established in other attribution domains, their formalization and implications for agent attribution remain largely underexplored. Prior works [2,3] typically adopt fixed attribution configurations and evaluate MAS performance after intervention, often implicitly treating the resulting scores as a universal measure of agent importance. Our contribution therefore lies in establishing and formalizing these design choices within a unified attribution framework for MAS, providing a foundation for systematically examining how they affect interpretation of agent importance.

Importantly, our formalization makes explicit why certain assumptions in prior work may not hold, while our experiments reveal their practical consequences. For example, attribution computed under one intervention b cannot be assumed to transfer reliably to another b', as each intervention defines a different coalition utility. Our experiments quantify the discrepancy between judge-estimated removal and actual agent ablation (Sec. 5.1), and demonstrate that using attribution computed under one intervention to guide another can lead to suboptimal agent selection (Sec. 5.4). Furthermore, our broader definition of intervention naturally accommodates any new protocols, e.g., prompt optimization, extending beyond the conventional removal/replacement paradigm in prior attribution work.

This unification also makes clear that agent importance is not universal but depends on the evaluation objective (Sec. 5.5). Additionally, holding other components fixed, we reveal an intriguing finding regarding attribution kernels: unlike observations in feature and data attribution [5,6], LOO can serve as an efficient and effective alternative to combinatorial methods, e.g., Shapley value, to guide intervention.

Together, we believe these findings demonstrate that our contribution goes beyond simply transferring established knowledge from other attribution domains to MAS, providing a foundation for practical guidance and new insights for agent attribution. We have revised the introduction, key takeaways, and results sections to make these practical implications more explicit.

Re W3: Additional baseline for attribution-guided prompt optimization (Sec. 5.4)

We thank the reviewer for this important suggestion. To clarify, our setup in Sec. 5.4 is to assess the effectiveness and transferability of attribution-guided rankings across intervention protocols for prompt optimization. In response to the reviewer’s suggestion, we consider two complementary comparisons: (1) interventions guided by non-attribution baselines, and (2) system-level optimization methods that go beyond individual agents.

For non-attribution baselines, we include random ranking and token-usage ranking, where agents are ranked by their token consumption in the full system (grand coalition) (Figure 14). We quantify intervention effectiveness using AUC (table below), showing that prompt LOO consistently outperforms both non-attribution baselines, whereas ablation-based ranking (CG-OPO) [2] can underperform random selection. Moreover, optimizing all agent prompts does not necessarily yield the best system performance. On PlanCraft and BrowseComp, optimizing top-2 agents can outperform jointly optimizing all agents (Figure 14). These findings highlight the effectiveness of attribution-guided intervention, e.g., prompt LOO, in identifying promising agents for targeted optimization and improving overall MAS performance.

Table 2: Intervention AUC of top-kk optimized prompt deployment

MethodPlanCraftWorkBenchBrowseComp-PlusLCBMean
Prompt LOO0.420 (0.004)0.361 (0.006)0.517 (0.004)0.652 (0.004)0.487 (0.002)
CG-OPO0.404 (0.004)0.322 (0.006)0.496 (0.004)0.628 (0.004)0.462 (0.002)
Random0.392 (0.004)0.327 (0.005)0.499 (0.004)0.637 (0.003)0.469 (0.002)
Token-usage-based0.389 (0.004)0.323 (0.006)0.483 (0.004)0.642 (0.004)0.464 (0.002)

For system-level optimization baselines, we include Meta-Harness [7], which optimizes MAS context and tool management, and AFlow [8], which optimizes system-level workflow orchestration. Both target broader MAS configurations rather than individual agents and are optimized under the same rollout budget as prompt optimization. The table below reports mean success rates on PlanCraft, averaged across topologies. Both prompt LOO and prompt Shapley outperform AFlow and Meta-Harness. This suggests that system-level optimization does not necessarily yield greater performance gains, highlighting the effectiveness of attribution-guided prompt optimization.

Table 3: PlanCraft success rate (%): Top-1 / Top-2 optimized

Method
Base39.3 (1.00)
CG-OPO40.0 (1.01) / 40.0 (1.01)
Prompt LOO41.3 (1.0) / 42.0 (1.01)
Random40.6 (0.93)/ 41.4 (0.96)
Token usage39.3 (1.00) /41.3 (1.01)
--------
MetaHarness0.395 (0.06)
AFLow0.385 (0.06)

Re W4: Agent interactions versus individual importance.

We thank the reviewer for raising this important and insightful point!

We agree that higher-order interactions can play a critical role in MAS, particularly in settings involving emergent social risks. However, we respectfully disagree that this diminishes the value of agent-level attribution. Our top-kk intervention experiments (Fig. 6 and Fig. 14, quantified by Table 2 above), together with our response to W1, demonstrate that targeted agent-level interventions can improve MAS performance while reducing inference costs, even in the presence of complex agent interactions, as seen in decentralized, centralized, and hybrid topologies. These findings highlight the practical value of identifying influential agents.

Additionally, agent-level attribution does not disregard interactions but reflects their effects on individual agents' contributions to system utility. As the reviewer noted, Shapley values account for interactions through marginal contributions across coalitions, while LOO measures each agent's marginal effect in the full system, capturing the net effects of its interactions without explicitly decomposing them. Thus, agent-level attribution remains informative even when system behavior depends on higher-order interactions. Moreover, explicitly estimating higher-order interactions can incur substantially greater computational costs in larger MAS [9–11]. Furthermore, our additional 10-agent experiments demonstrate LOO's effectiveness in identifying key players and improving performance under constrained evaluation budgets (see our response to W1 by reviewer h69T). We believe that these observations further reinforce the practical importance of agent-level attribution.

Reference


[1] MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems? arXiv 2026

[2] Xia, Yihan, et al. "Hivemind: Contribution-guided online prompt optimization of llm multi-agent systems." AAAI 2026

[3] Ebrahimi, et al. "An adversary-resistant multi-agent llm system via credibility scoring." ACL 2025.

[4] Erol, Mehmet Hamza, et al. "Cost-of-pass: An economic framework for evaluating language models." ICLR 2026

[5] Covert, et al. "Understanding global feature contributions with additive importance measures." NeurIps 2020

[6] Ghorbani, et al. "Data shapley: Equitable valuation of data for machine learning."ICML, 2019.

[7] Lee, Yoonho, et al. "Meta-harness: End-to-end optimization of model harnesses." arXiv 2026

[8] Zhang, Jiayi, et al. "Aflow: Automating agentic workflow generation." ICLR 2025

[9] Muschalik, Maximilian, et al. "shapiq: Shapley interactions for machine learning." NeurIPS 2024

[10] Sundararajan, et al.. "The shapley taylor interaction index." ICML 2020

[11] Butler, Landon, et al. "Proxyspex: Inference-efficient interpretability via sparse feature interactions in llms." NeurIPS 2025.

Reviewer_h96TOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper formalizes agent attribution in multi-agent LLM systems as a cooperative game with three design choices: the intervention protocol, the coalition distribution, and the target metric. It compares four protocols (agent ablation, introspective removal by an LLM judge, model replacement, and prompt optimization) and four coalition distributions (Leave-One-Out, Shapley, Owen, and Myerson) across four benchmarks and four communication topologies. It reports that introspective removal does not reproduce ablation utilities, that LOO ranks agents about as well as the combinatorial methods at a fraction of the cost, that prompt-conditioned attribution picks better prompt-optimization targets than ablation-based attribution. The authors report that in a medical system the agents that help diagnostic accuracy differ from those that help ethical alignment.

Critical strengths
  1. The framework separates design choices that prior work fixed together, and the experiments vary one at a time. Figure 9 shows the same system under four protocols, and the sign pattern differs by protocol. This is consequential because it explains why prior attribution results disagree and tells a practitioner which question each method answers.

  2. The introspective-removal result is a useful negative result. Judge-estimated coalition utilities diverge from executed ones across four judges, three judge-enhancement variants, and all topologies (Tables 1, 3, and 4), and judge-ranked removal at k equal to one lowers success in most settings (Appendix E.5). This is consequential because judge-based attribution is the cheap option the field is likely to adopt.

  3. The evaluation is broad and the reporting is thorough. Four benchmarks, four topologies, four protocols, alternative backbones, five substitute models, paired confidence intervals, and full token and cost accounting (Table 9), with prompts and code released.

Critical weaknesses
  1. The LOO claim is tested only where exact Shapley is cheap. All systems have four to six agents, so Shapley costs at most 63 evaluations, and the reported cost ratio of up to twelve times is measured in that regime. The case for LOO is the large-agent regime, which is untested. The paper's own analysis shows that LOO agreement depends on an essential agent, falls by 0.11 when that agent is held fixed, and drops to 0.10 to 0.50 under redundancy on WorkBench (Section 5.2). A sampled Shapley estimator at LOO's budget (Castro et al., 2009) is the natural competitor and is absent. The authors ought to add one system with ten or more agents, compare LOO with permutation-sampled Shapley at matched cost, and scope the claim to the conditions under which it holds.

  2. The intervention results have no non-attribution control. Sections 5.3 and 5.4 and Figures 5(c), 6, and 12 compare rankings from four attribution methods, which produce nearly identical intervention curves. Without a random ranking or a token-usage ranking on the main benchmarks, the reader cannot tell whether attribution guides the intervention or whether any order would do. Only the MDTeamGPT audit includes such a control (Figure 7d). The authors ought to add random and heuristic rankings to Figures 5(c) and 6 and report the paired difference.

  3. The medical audit rests on one substitute model and small samples. The attribution is LOO under model replacement with one substitute, Qwen 3.5 27B, on 100 MedQA and 200 MedEthicsQA items. Appendix E.8 shows that replacement attribution and rankings change with the substitute. The reported effects are a four-point ethics gain and a near-zero accuracy change, and the open-ended ethics items are graded by an LLM judge whose agreement with humans is not reported. The authors ought to repeat the audit with a second substitute, report confidence intervals for the rankings, and validate the judge on a sample.

Questions for authors
  1. How does LOO compare with permutation-sampled Shapley at the same token budget, and does the agreement hold on a system with ten or more agents?

  2. What intervention AUC does a random ranking achieve on the four main benchmarks, relative to the attribution-guided rankings in Figure 5(c)?

  3. Does the MDTeamGPT ranking reversal between accuracy and ethics hold with a second substitute model?

4 comments

Dear reviewer h96T

We thank the reviewer for their thoughtful evaluation and constructive feedback. We particularly appreciate the reviewer’s recognition of our work as “consequential because it explains why prior attribution results disagree and tells a practitioner which question each method answers.” We are also happy to hear that the reviewer highlighted our findings on introspective removal as “a useful negative result” and acknowledged that “the evaluation is broad and the reporting is thorough.” Below, we address the reviewer’s concerns and welcome any further questions.

Re: LOO vs. Shapley value in larger multi-agent systems under matched budgets (W1 & Q1)

We thank the reviewer for this constructive suggestion. To address the concern about scalability and missing matched-budget comparisons, we have extended our experiments to a 10-agent system where we intentionally include two redundant worker agents on WorkBench, comparing LOO against permutation-sampled Shapley [1] under the same budget of 11 coalition evaluations (n+1)(n+1), alongside random and token-usage baselines. Below, we report intervention AUC and 95%CI.

Table 1: Intervention AUC (agent ablation):

ArchLOOShapleyRandomToken usage
independent0.238 (0.02)0.221 (0.02)0.185 (0.01)0.189 (0.02)
decentralized0.200 (0.02)0.188 (0.02)0.142 (0.01)0.131 (0.01)
centralized0.162 (0.01)0.149 (0.01)0.146 (0.01)0.151 (0.02)
hybrid0.167 (0.01)0.160 (0.01)0.128 (0.02)0.144 (0.01)

Table 2: Intervention AUC (model replacement):

ArchLOOShapleyRandomToken-usage-based
Ind.0.313 (0.01)0.322 (0.01)0.256 (0.00)0.234 (0.02)
Dec.0.257 (0.01)0.235 (0.01)0.220 (0.01)0.202 (0.02)
Cent.0.277 (0.01)0.271 (0.00)0.232 (0.00)0.238 (0.02)
hybrid0.293 (0.01)0.253 (0.02)0.210 (0.02)0.233 (0.02)

Table 3: Intervention AUC (prompt optimization):

ArchLOOShapleyRandomToken-usage-based
Ind.0.520 (0.02)0.502 (0.02)0.472 (0.01)0.449 (0.01)
Dec.0.407 (0.01)0.404 (0.01)0.339 (0.01)0.324 (0.01)
Cent.0.328 (0.009)0.305 (0.007)0.296 (0.008)0.297 (0.02)
hybrid0.355 (0.008)0.333 (0.005)0.328 (0.004)0.348 (0.009)

Under the same evaluation budget, LOO achieves higher intervention AUC than sampled Shapley across all four topologies and intervention protocols. Notably, sampled Shapley sometimes fails to outperform the random and token-usage baselines under this constrained budget, whereas LOO consistently outperforms both baselines. These results extend our findings to larger systems, further demonstrating LOO's effectiveness and computational efficiency when accurate estimation of combinatorial attribution becomes increasingly costly. We’ve incorporated this additional evaluation in Appendix E.10 in the revised manuscript.

Re: Performance of control baselines in top-kk interventions (W2 & Q2)

We thank the reviewer for suggesting control baselines to further validate the effectiveness of attribution-based rankings. In response, we have added two baselines: (1) random ranking and (2) token-usage ranking, which ranks agents by their token consumption in the grand coalition.

The table below further quantifies the intervention performance of attribution-based methods and control baselines on PlanCraft, as shown in Figure 5(c). LOO achieves performance comparable to Shapley while consistently outperforming both control baselines across three intervention protocols. Specifically, LOO achieves relative paired-AUC improvements of 27.4% (+0.064), 20.3% (+0.064), and 14.5% (+0.053) over random ranking, and 11.2% (+0.030), 19.6% (+0.062), and 14.8% (+0.054) over token-usage ranking, respectively. Values are ordered by agent ablation, model replacement, and prompt optimization. These results demonstrate that attribution-guided rankings identify more effective intervention targets than random and token-usage rankings.

Table 4: PlanCraft intervention AUC (averaged across topologies).

InterventionLOOShapleyRandomToken usage
Agent ablation0.298 (0.01)0.295 (0.01)0.234 (0.01)0.268 (0.01)
Introspective removal0.419 (0.01)0.426 (0.01)0.392 (0.01)0.357 (0.01)
Model replacement0.379 (0.02)0.392 (0.02)0.315 (0.02)0.317 (0.01)
Prompt optimization0.418 (0.02)0.423 (0.02)0.366 (0.02)0.365 (0.01)

This finding holds across datasets and intervention protocols (Figures 20–22), where LOO consistently outperforms rankings based on the control baselines in identifying effective intervention targets.

In contrast, rankings derived from introspective removal show weaker performance when evaluated through actual agent ablation (Figure 20). On PlanCraft, averaged across topologies, introspective removal achieves a mean AUC of 0.259, only slightly higher than random ranking (0.234) and lower than token-usage ranking (0.268). These results further reinforce our negative findings on introspective removal.

We have incorporated these baselines into Figure 5(c) of the revised main text and Figures 20–22 of the appendix, extending the intervention experiments in Figure 6.

Re: Human–LLM agreement on MedEthicsQA and ranking reversals across models (W3 & Q3)

We thank the reviewer for raising these important questions.

Regarding human–LLM agreement, our open-ended ethics evaluation follows the protocol established in [2], which reports 96.5–97.5% agreement between LLM judgments and human annotations.

Regarding ranking reversals across models, we extend our original analysis (Sec. 5.4) of Qwen3.5-27B to three additional models (Qwen3.5-9B, Qwen3.5-35B, and Gemma4-26B), as shown in the newly added Figure 23. We report Spearman rank correlations (ρ\rho) across runs between agent attribution rankings on MedQA and MedEthicsQA for all four models:

ModelSpearman ρ (95%CI)
Gemma4-26B−0.89 [−0.98, −0.78]
Qwen3.5-9B−0.89 [−0.95, −0.66]
Qwen3.5-27B−0.95 [−1.00, −0.85]
Qwen3.5-35B+0.74 [0.53, 0.88]

The results reveal strong ranking reversals in two of the three newly evaluated models, with Gemma4-26B and Qwen3.5-9B exhibiting strongly negative rank correlations (−0.89), consistent with our original finding for Qwen3.5-27B (−0.95). In contrast, Qwen3.5-35B shows a positive correlation (+0.74), indicating that ranking reversals are model-dependent rather than universal. These findings reinforce our key takeaway that agent attribution depends on the underlying model and intervention protocol (Sec. 5.2), as well as the evaluation objective (Sec. 5.5). We have incorporated this analysis, along with additional intervention results across substitute models (Table 12), into Appendix F.

Reference


[1] Castro, Javier, Daniel Gómez, and Juan Tejada. "Polynomial calculation of the Shapley value based on sampling." Computers & operations research 2009.

[2] Wei, Jianhui, et al. "Medethicsqa: A comprehensive question answering benchmark for medical ethics evaluation of llms." arXiv 2025.

Thank you for the detailed response. The new experiments answer all three of my questions. I will raise my score accordingly.

Re: LOO at scale. The 10-agent WorkBench system with redundant workers is the right stress test, and comparing against permutation-sampled Shapley at LOO’s budget answers my question directly.

Re:control baselines. the random and token-usage rankings in Figure 5(c) and Figures 20 to 22 show that attribution guides the intervention.

Re: substitute. The three added substitutes are informative. For two of them, the agent rankings for accuracy and for ethics alignment are negatively correlated, as with the original substitute. For Qwen3.5-35B they are positively correlated. Section 5.5 currently states the same pattern holds for the other substitutes. The authors might clarify it holds for two of the three, as the response states.

Dear reviewer h96T,

Thank you for your positive feedback and support for clear acceptance. We have revised the statement regarding the additional substitute models in Section 5.5 (Lines 465–466) of the manuscript accordingly.

Thank you again for your constructive feedback and support.

Best regards,
The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revision summary

Dear reviewers

We thank the reviewers for their thoughtful and constructive feedback. We appreciate their recognition of our framework's ability to disentangle attribution design choices, the practical significance of our empirical findings—including the limitations of introspective removal—and the breadth and rigor of our experimental evaluation [h96T, CaCQ].

In response to the reviewers' concerns, we have provided clarification and conducted additional experiments, all of which are detailed in individual responses. We summarize below:

Strengthened experimental validation.

  • We add random ranking and token-usage ranking baselines to our top-kk intervention experiments (Fig. 5(c) for PlanCraft; Figs. 20–22 for all datasets) to validate attribution-guided agent selection [h69T, CaCQ, ZDig]. We extend our evaluation to a larger, 10-agent system across intervention protocols and further validate LOO through comparisons with Shapley values under equal coalition budgets [h69T]. We broaden our evaluation of attribution-guided optimization by incorporating MetaHarness and AFlow, two additional methods that support optimization beyond individual agents [ZDig]. Finally, we extend our MDTeamGPT experiments (Sec. 5.5) to three additional substitute models—Gemma 4 26B, Qwen3.5 9B, and Qwen3.5 35B—to demonstrate further how attribution ranking reversals across evaluation objectives depend on the underlying model [h69T].

Clarified practical implications and generalizability.

  • We clarify the importance of agent-level attribution, the setup and performance–cost benefits of selective optimized prompt deployment, and our contributions relative to the existing attribution field [ZDig]. We provide further evidence that attribution methods identify effective intervention targets [CaCQ], and refine our actionable guidelines for selecting attribution methods and intervention protocols [CaCQ, ZDig]. We also demonstrate their applicability to another MAS architecture, MetaGPT, where attribution-guided interventions identify agents for optimization and improve system performance [CaCQ].

We revised the introduction, key takeaways, figures, and results section and added experiments to the appendix. The revised manuscript incorporates the clarifications and additional results detailed in our responses to individual reviewers. We believe our responses comprehensively address the reviewers’ concerns and ensure these clarifications and additional results are included in our revised manuscript.

We look forward to your response and are happy to address any further questions!

Dear ACs and Reviewers

As this discussion period draws to a close, we would like to express our sincere appreciation to the Area Chair and all reviewers for their time, thoughtful evaluations, and constructive feedback. We are particularly grateful to Reviewers h96T and CaCQ for recognizing the contributions of our work, to Reviewer CaCQ for encouraging us to make our insights more accessible and actionable to readers beyond domain experts, and to Reviewer ZDig for clarifying the importance of demonstrating practical benefits relative to alternative approaches and setups. These discussions have helped us better articulate the practical value of our work and identify opportunities for translating our findings into real-world applications for agentic research.

This rebuttal discussion has been genuinely meaningful to us. The reviewers' thoughtful questions and feedback have encouraged us to reflect more deeply on the broader implications, practical applications, and future directions of our research, as well as how to communicate our findings in ways that enable a wider audience to benefit from these insights. We once again thank the Area Chair and all reviewers for the time and expertise they have devoted to our work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.