acceptodds
Under review as a conference paper at ICLR 2027

Disentangling Agent Contribution in LLM Multi-Agent Systems through Controlled Interventions

Abstract

Identifying which agent actually drives the success of a large language model (LLM)-based multi-agent system remains an open challenge. Aggregate performance cannot answer this question, and our results show that attribution does not simply identify the most capable reasoner. We introduce MASContributionBench, covering seven architectures and seven task suites, and combine leave-one-out (LOO), sampled Shapley, and sampled Banzhaf attribution with controlled interventions on topology, role prompts, and runtime permissions. Complete MAS outperform single-agent and solo-role baselines, while the aggregate difference from random teams remains unresolved; collaboration gains alone therefore do not validate a prescribed organization. More strikingly, changing validated static predecessor sets reverses an unchanged position's mean LOO sign with modest team-utility change. Exchanging functional prompts moves credit to unedited roles without improving the team, demonstrating that contribution does not simply follow role function. Changing Coder write access leaves of outcomes unchanged, while granting final-answer authority to Verifier produces no Verifier-selected outputs in all runs. These findings reveal that measured contribution conflates useful reasoning, routing leverage, and control of the evaluator-visible output. Contribution therefore measures the counterfactual fragility of an execution protocol, rather than an intrinsic property of an agent or role.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM-based multi-agent systems, agent contribution attribution, multi-agent evaluation, Shapley value, counterfactual analysis, controlled interventions, communication topology, role attribution

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tsr5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MASContributionBench, a benchmark covering seven multi-agent system (MAS) architectures, seven tasks, three attribution methods (leave-one-out, Shapley, and Banzhaf), and three interventions (topology, role prompts, and runtime permissions). With results from MASContributionBench, the authors make the following observations.

  1. A random team (MAS) can perform similarly as a pre-defined MAS configuration, and different topologies have similar performance on SWE-Bench (verified). These results suggest that aggregate performance does not provide evidence that one MAS architecture is better than the other.
  2. Leave-one-out (LOO) attribution has different score distributions compared to Shapley and Banzhaf attributions, and scores are context-dependent.
  3. Across interventions based on topology, role prompts, and runtime permissions, results show that contribution can change even for the non-intervened agents, suggesting the fluidity of how different agents are intended to contribute.
Critical strengths
  • The benchmark covers a wide range of tasks, MAS topologies, attribution methods, and intervention methods. I hope the authors can release the traces and performance data points for future work to build upon.
  • The results in Section 3.4 are insightful in my opinion. Across interventions based on topology, role prompts, and runtime permissions, results show that contribution can change even for the non-intervened agents, suggesting the fluidity of how different agents are intended to contribute. In other words, these results show that we cannot reason which agents are important simply based on the null topology, roles, and permissions. I think this framing is worth highlighting more in the paper.
Critical weaknesses
  • Although this paper provides insights, these insights are not demonstrated in downstream use cases such as MAS optimization (e.g. by removing unnecessary agents or removing unnecessary communication edges).
  • The differences between LOO and coalition-based attributions such as Shapley and Banzhaf have already be observed in prior work [1].

References [1] Lu et al. (2026) Agents that matter: Optimizing multi-agent LLMs via removal-based attribution.

Questions for authors
  1. Do the insights observed from your results actually lead to a better optimized MAS configurations?
  2. The percentages of resolved tasks in SWE-Bench (Verified) are low across the topologies. Could this be due to the low capability of Qwen3.8-27B-FP8? In other words, it is possible that the similarity between different topologies does not hold for an LLM more capable at coding.
  3. Do the insights observed from your paper hold using a different LLM?
8 comments
AuthorsOpenReview ↗

Summary Response

Dear Reviewer tsr5,

We thank you for recognizing the breadth of our benchmark and the insights from our intervention results. We also appreciate your suggestion to highlight how the measured contribution of an unchanged agent can vary with the surrounding system.

Distinction from Agents that Matter. We agree that differences between LOO and coalition-based attributions have been observed in prior work and do not claim this general observation as novel. Our work complements it through controlled interventions and execution audits, examining how routing, role prompts, and permissions affect measured contributions and how configured interventions are realized during execution. (Weakness 2)

We summarize our responses to your questions as follows:

  1. We conduct a preliminary single-agent pruning study to examine potential performance–cost trade-offs. The results identify candidates for further testing and illustrate why null-replacement attribution does not directly predict removability. The candidates have not yet been validated on new tasks. (Weakness 1; Response 1/3)

  2. We evaluate three additional backbones on SWE-bench Verified, including a coding-oriented model and Claude Opus 4.5. We observe low confirmed completion rates under our scaffold and clarify the limitations of comparing architectures in this setting. (Response 2/3)

  3. We examine the Verifier topology effect in homogeneous teams with additional backbones and a heterogeneous model team. The results support context-dependent attribution in the tested settings, while the observed sign reversal and its magnitude are not universal. (Response 3/3)

We will clarify the scope of our findings and highlight the interpretation of our intervention results in the paper. We hope our responses address your concerns and look forward to further discussion with you.

AuthorsOpenReview ↗

Response 1/3

  • Single-agent pruning reveals potential performance–cost trade-offs that require further validation.

To examine whether our findings can inform team configuration, we conducted a preliminary pruning study on the A3 DAG. We compared the complete team with five modified teams, each obtained by removing one eligible intermediate agent: Planner, Retriever, Coder, Debugger, or Tester. For each removal, we connected the agent's predecessors to its successors and retained the Finalizer.

This study evaluates single-agent pruning candidates only. It does not evaluate joint removal of multiple roles or a fully redesigned team configuration. All conditions were evaluated on the same discovery tasks and seeds: 30 GPQA Diamond tasks and 30 IFBench tasks, with three seeds per task.

Table 1 reports utility and average total token use. Utility is normalized exact-match accuracy on GPQA and strict prompt-level instruction-following pass rate on IFBench. For each displayed candidate, the random-removal comparator is the uniform average over the other four eligible single-agent removals.

Table 1: Preliminary single-agent pruning results on discovery tasks. ΔU\Delta U is the utility difference relative to the complete team, in percentage points (pp). For each displayed candidate, the paired 95% confidence interval is computed by task-cluster bootstrap, retaining all three seeds and compared conditions for each sampled task. Tokens are average input and output tokens per run.

BenchmarkTeam configurationUtilityΔU\Delta U95% CI for ΔU\Delta UTokens
GPQAComplete team50.00%——19,165
GPQARemove Debugger61.11%+11.11+11.11 pp[+4.44,+20.00][+4.44,+20.00] pp13,625
GPQARandom: other four55.28%+5.28+5.28 pp—16,728
IFBenchComplete team89.66%——7,747
IFBenchRemove Coder89.66%0.000.00 pp[−4.60,+4.60][-4.60,+4.60] pp6,654
IFBenchRandom: other four85.92%−3.74-3.74 pp—6,252

On GPQA, removing the Debugger increased observed utility by 11.11 percentage points and reduced average token use by 28.9% relative to the complete team. Its observed utility was 5.83 points above the average for removing one of the other four roles (paired 95% CI: [+2.22,+10.56][+2.22,+10.56] points). On IFBench, removing the Coder produced the same observed utility as the complete team while reducing average token use by 14.1%. Its utility was 3.74 points above the other-four random-removal average, but the paired 95% CI was [−1.15,+9.48][-1.15,+9.48] points. Moreover, its token use was higher than that random-removal average. Thus, the IFBench evidence does not establish an advantage over random removal.

These random comparisons concern specific removal candidates selected after inspecting the discovery results. They do not constitute an end-to-end comparison between an attribution-guided selection strategy and a random selection strategy. Indeed, the GPQA results expose a concrete limitation of the original null-replacement attribution. Defining LOO as complete-team utility minus intervened-team utility, the Debugger's null-replacement LOO was +1.11+1.11 points, whereas its real-removal LOO was −11.11-11.11 points. Ranking roles by the lowest null-replacement LOO therefore did not place the Debugger among the leading removal candidates, even though its real removal had the highest observed utility. Null replacement retains the agent's graph position, while real removal also changes communication paths; the former should not be treated as a direct prediction of the latter.

The GPQA candidate also requires a budget-controlled follow-up. The complete team's Finalizer reached its 4,096-token output limit in 38 of 90 runs, compared with 29 of 90 after Debugger removal. All ten paired runs that improved after removal had reached the limit in the complete-team condition. Among the 51 paired runs, spanning 19 tasks, in which neither condition reached the limit, both conditions passed 45 runs. This post-hoc subset comparison is descriptive: conditioning on outcomes observed after both interventions can introduce selection bias and does not isolate the causal effect of the output budget. The GPQA utility advantage should therefore not yet be treated as a validated improvement.

Both candidates in Table 1 were identified after inspecting the discovery results and have not been tested on independent tasks. The present study therefore provides testable pruning candidates and evidence about the limits of null-replacement attribution, but it does not yet establish an attribution-guided configuration improvement on new tasks.

AuthorsOpenReview ↗

Response 2/3

  • SWE-bench Verified: backbone capacity and architectural similarity.

To examine whether the low completion rates and apparent similarity across architectures are specific to Qwen3.8-27B-FP8, we evaluated three additional backbones under our experimental scaffold. We first tested Gemma-4-31B-it-QAT, a different open-weight model, and observed similarly low completion rates. After analyzing the failure cases, we evaluated Qwen3-Coder-30B-A3B-Instruct-FP8, a coding-oriented model represented on the SWE-bench leaderboard. We further evaluated Claude Opus 4.5 as a stronger closed-weight backbone.

For each backbone, we evaluated A2 Chain and A3 DAG on the same 30 task instances, with three runs per instance. The table reports confirmed resolved runs out of 90 planned runs per condition.

BackboneArchitectureConfirmed resolved / PlannedRate
Qwen3.8-27B-FP8A2 Chain5 / 905.6%
Qwen3.8-27B-FP8A3 DAG4 / 904.4%
Gemma-4-31B-it-QATA2 Chain3 / 903.3%
Gemma-4-31B-it-QATA3 DAG6 / 906.7%
Qwen3-CoderA2 Chain3 / 903.3%
Qwen3-CoderA3 DAG3 / 903.3%
Claude Opus 4.5A2 Chain17 / 9018.9%
Claude Opus 4.5A3 DAG14 / 9015.6%

Each condition comprises 30 distinct tasks and three runs per task. Rates use all 90 planned runs as the denominator and report confirmed resolved runs.

The results suggest that low completion is not unique to Qwen3.8-27B-FP8 under our scaffold: both Gemma and the coding-oriented Qwen3-Coder also produced few confirmed solutions. Claude achieved more confirmed resolutions than the other evaluated backbones in both architectures, indicating that backbone choice matters in this setting. Nevertheless, its confirmed completion rates remained below 20%.

For Claude, Chain and DAG resolved 17 and 14 of the 90 planned runs, respectively, a difference of 3.3 percentage points. Thus, we still observe a small absolute gap between these two architectures with this closed-weight backbone. However, this observation does not establish architectural equivalence: the sample is limited, some outcomes remain unknown, and A2 and A3 differ in their role sets as well as their communication structures.

Our failure analysis shows that during generation, agents received only the issue description and could not browse the repository, edit files, or execute tests. A substantial fraction of submitted patches could not be applied to the target repository; thus, completion rates reflect pipeline constraints as well as backbone behavior, and should not be read as standalone coding ability.

These findings address the concern in a bounded setting: replacing Qwen3.8 with other backbones, including a coding-oriented model and Claude Opus 4.5, did not eliminate low completion rates, and the absolute Chain–DAG gap remained small for Claude. We cannot rule out that a stronger model combined with repository access and iterative test feedback would reveal differences that are difficult to observe here. We therefore do not generalize the observed similarity to tool-enabled coding agents or attribute it solely to topology.

SWE-bench Verified nevertheless provides a useful repository-level evaluation: it tests whether outputs from a multi-agent workflow can become patches that apply to real repositories and pass their tests. In our setting, it exposes a gap between textual collaboration and an executable software artifact. We interpret this finding as a limitation of the current execution setting and as motivation for future repository-aware, tool-enabled comparisons, rather than as a definitive ranking of architectures.

Reviewer_9GtdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to measure individual agents’ contributions in LLM multi-agent systems. The authors compare leave-one-out, Shapley, and Banzhaf scores under changes to communication structures, role prompts, and permissions. The results show that measured contribution depends on an agent’s position and execution rules, as well as its ability.

Critical strengths
  • I like the focus on what contribution scores actually tell us. This is useful for understanding a team beyond its overall performance.

  • The authors carefully check whether the interventions work as intended and report cases where the actual behavior differs.

Critical weaknesses
  • The contribution over "Agents that Matter: Optimizing Multi-Agent LLMs via Removal-Based Attribution" needs more explanation. That paper already studies how attribution depends on coalition sampling, removal rules, and evaluation metrics. What do the additional interventions in this work help us understand that was missing before?

  • Some permission results seem to follow directly from how the system is implemented. In A5, Verifier receives final-answer permission, but the system still uses Supervisor’s answer in every run. This shows that permission alone does not give an agent control, but tells us little about whether Verifier could actually take over the role.

  • I am not sure the experiments fully separate reasoning ability from information flow and output control. Replacing an agent’s response with a null response removes its reasoning, but also changes the input received by later agents. These effects remain mixed together.

Questions for authors
  • What happens if the system actually uses the verifier’s answer after transferring permission? I think a comparison with Finalizer and Supervisor would make this experiment more informative.

  • Appendix A.2 describes bypass removal, but the reported attribution results all use null replacement. Could you compare both approaches, especially for agents that relay information or provide the final answer?

  • Most primary benchmarks use only thirty tasks. Could you check the main findings with more tasks and another model, especially cases where an agent’s contribution changes from positive to negative?

  • Could these findings guide improvements to a team? For example, can you identify and remove a redundant agent without reducing performance on new tasks?

9 comments
AuthorsOpenReview ↗

Summary Response

Dear Reviewer 9Gtd,

We thank you for recognizing our focus on interpreting agent contribution and our careful validation of the interventions. We also appreciate your insightful questions on the distinction from prior work, removal protocols, and the practical use of attribution.

Distinction from Agents that Matter. We agree that prior work has shown that attribution depends on coalition sampling, removal protocols, and evaluation metrics. Our work complements these findings through controlled interventions and execution audits: we examine how routing changes the attribution of an unchanged position, how prompt changes redistribute credit, and how configured authority can differ from the output source actually selected by the executor. These results help interpret attribution in the context of information flow and execution rules. (Weakness 1)

We summarize our responses to your questions as follows:

  1. We conduct a comparison in which Finalizer, Supervisor, or Verifier actually supplies the final answer. Verifier achieves competitive observed utility, although the results do not establish a clear overall advantage. (Weakness 2; Response 1/4)

  2. We compare null replacement and bypass removal on A2. The differences highlight the sensitivity of attribution to the removal protocol, while neither intervention fully separates reasoning from information flow and output selection. (Weakness 3; Response 2/4)

  3. We evaluate the Verifier topology effect with two additional backbones and additional disjoint tasks. The results support context dependence within the tested settings, although the sign reversal is not universal across backbones. (Response 3/4)

  4. We conduct a preliminary single-agent pruning study with utility, token cost, and random-removal comparisons. The results identify candidates for further testing, but do not yet establish performance preservation on new tasks. (Response 4/4)

We have refined our interpretation and clarified the limitations of these results. We hope our responses address your concerns and look forward to further discussion with you.

AuthorsOpenReview ↗

Response 1/4

  • Verifier output after permission transfer.

We examined whether transferring final-answer authority actually causes the system to submit Verifier's answer. In the original A5 permission-transfer experiment, Verifier received the final_answer permission, but the router selected Supervisor as a fallback in all 450 runs. Thus, the original result did not measure the system's performance when Verifier actually supplied the final answer.

We therefore conducted a new system-level comparison with three conditions: Finalizer, Supervisor, or Verifier is granted final-answer authority and selected as the output source. Each condition runs the complete A5 system with the same model, task set, topology, and seeds. The authorized output source and the actual submitted source are recorded for every run. If the selected role does not provide an available output, the system does not silently fall back to another role.

We excluded one instruction-following task from all three conditions because its letter-frequency constraint supplied ! to a checker that accepts letters only; the checker otherwise selected a random letter. The resulting comparison contains 149 tasks and three seeds, yielding 447 paired runs per condition. We re-scored the recorded outputs after consistently extracting each selected role's artifact; unavailable or unparseable artifacts were assigned a score of zero. This re-scoring also avoids extracting an answer from the surrounding explanation rather than from the submitted artifact.

Table 1: System-level comparison of final-answer sources in A5. Each condition contains 447 task–seed runs.

Final-answer sourceArtifact availabilityStrict utilityIFBench pass rate
Finalizer72.93%45.86%90.80%
Supervisor73.60%46.76%86.21%
Verifier99.33%49.44%83.91%

As shown in Table 1, when the system actually submits Verifier's output, its overall strict utility is 49.44%, compared with 45.86% for Finalizer and 46.76% for Supervisor. Verifier also produces a parseable, non-empty artifact more frequently. However, on the IFEval-derived subset, Verifier's strict pass rate is lower than those of the other two sources. The overall ordering therefore does not hold uniformly across datasets, and the strict-utility comparison includes differences in artifact availability.

We additionally computed paired differences over the same task–seed runs. The Verifier–Finalizer difference is +3.58 percentage points (95% bootstrap CI: [-0.22, 7.61]), and the Verifier–Supervisor difference is +2.68 percentage points (95% bootstrap CI: [-0.67, 6.26]). Bootstrap resampling is stratified by dataset and performed at the task level, retaining the three seeds for each sampled task. Both intervals include zero, so these results do not establish a clear overall advantage for Verifier. Against Finalizer, Verifier wins, ties, and loses on 39, 385, and 23 task–seed pairs, respectively; against Supervisor, the corresponding counts are 32, 395, and 20.

These results clarify the distinction between receiving authority and actually supplying the final answer. The original transfer condition selected Supervisor through fallback. Under the new strict routing intervention, Verifier's output actually reaches the evaluator and obtains competitive overall utility, but the observed differences do not support a conclusive claim that it is better than Finalizer or Supervisor.

AuthorsOpenReview ↗

Response 2/4

  • Null replacement versus bypass removal.

We conducted a matched comparison on the A2 Chain architecture using GPQA Diamond and IFBench (30 tasks per benchmark and 3 seeds). For each task, seed, and agent position, we computed LOO attribution under both null replacement and bypass removal, yielding 720 matched task–seed–role comparisons. The complete-team utility was identical across the two protocols for all matched comparisons. Confidence intervals below use a paired bootstrap over tasks, retaining all three seeds within each sampled task.

AgentNull LOOBypass LOODifference [95% CI]
Planner0.0220.022+0.000 [-0.044, +0.039]
Coder0.0670.050-0.017 [-0.078, +0.039]
Reviewer0.0890.017-0.072 [-0.117, -0.028]
Finalizer0.7830.411-0.372 [-0.478, -0.272]

Difference is Bypass LOO minus Null LOO. Values are averaged equally across the two benchmarks.

For the intermediate Coder, the difference was small and its confidence interval included zero. In contrast, the Reviewer's attribution decreased from 0.089 under null replacement to 0.017 under bypass removal. This result is consistent with part of the null-replacement effect reflecting disrupted information flow: bypass allows the Reviewer's predecessor to communicate directly with its successor. However, because bypass also changes the graph and the downstream input, the difference does not identify a pure information-transfer effect.

The Finalizer shows the largest difference. Under null replacement the removed-team utility was 0.000; under bypass removal it was 0.372. In all 180 Finalizer-bypass runs, the executor selected the upstream Reviewer's output; under the bypass protocol the removed position no longer mediates the authorized output. This contrast therefore includes a change in output selection and should not be interpreted as the Finalizer's reasoning contribution alone, nor as a strict authority-transfer experiment.

We will revise the interpretation accordingly: null-replacement LOO measures a position's dependence under the specified execution protocol, potentially combining content generation, information mediation, and output selection. Bypass removal defines a different counterfactual and helps reveal this sensitivity, but neither comparison fully isolates reasoning ability from information flow.

Reviewer_SndgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies agent contribution in LLM-based multi-agent systems (MAS), defined as counterfactual marginal utility under a specified execution context. It introduces MASCONTRIBUTIONBENCH: seven MAS architectures (PEV, Chain, DAG, MetaGPT-lite, Star, Debate, Graph) evaluated on seven task suites (AIME 2026, GPQA Diamond, HLE, ARC-AGI-2, IFBench, LiveCodeBench, SWE-bench Verified) with matched single-agent, solo-role, and random-team baselines, all running a single pinned open-weights backbone (Qwen3.8-27B-FP8) under fixed tasks, seeds, decoding, and evaluation protocol. The benchmark combines three attribution estimators — leave-one-out with null-agent replacement, sampled Shapley (K=8), and sampled Banzhaf (K=8) — with controlled interventions on communication topology (runtime-validated Chain–DAG rerouting), functional role-prompt swaps, and runtime-enforced permissions (write solution / final answer, revocation and transfer). A validity gate in the utility definition separates solution quality from authority to expose a solution to the evaluator.

Key findings: (1) complete MAS outperform single-agent and solo-role baselines but are statistically indistinguishable from random five-role teams (Δ = −0.0002, 95% CI [−0.031, +0.031]); (2) LOO, Shapley, and Banzhaf measure distinct estimands with only tier-level agreement; (3) validated rerouting reverses an unchanged position’s mean LOO sign (+0.051 → −0.040) with modest team-utility change, and prompt swaps redistribute credit — including to unedited roles (Supervisor +0.176) — without improving the team; (4) revoking Coder write access leaves 87.8% of outcomes unchanged, while transferring final-answer authority to Verifier yields 0/450 Verifier-selected outputs. The central claim: measured contribution reflects the counterfactual fragility of an execution protocol, not an intrinsic property of an agent or role.

Critical strengths
  1. Exemplary experimental control and transparency. All comparisons hold a single pinned open-weights backbone (Qwen3.8-27B-FP8, with revision hash), tasks, seeds, decoding, and evaluators fixed; permissions are enforced at runtime rather than through prompt instructions; realized executions are audited (Chain–DAG predecessor sets verified at all 4,500 node inputs, and the configured-Star condition is honestly demoted to an executor diagnostic when it fails to realize its design); uncertainty is reported via paired task-cluster bootstrap intervals; exact Shapley is computed for A1 as a calibration of the K=8 sampler; and an anonymized repository is released. This level of hygiene is rare in MAS evaluation and materially raises my confidence in the paper’s negative and causal findings — it is the main reason I lean toward acceptance.

  2. A real and practically important interpretive contribution. The paper disentangles useful reasoning, routing leverage, and control of the evaluator-visible output, and shows with controlled evidence that attribution scores conflate them: sign reversal of an unchanged position under validated rerouting; credit redistribution without utility, including to unedited roles; and nominal authority failing to translate into realized control (0/450 Verifier-selected outputs after the authority transfer). The matched random-team null result (Δ = −0.0002) further deflates the common practice of justifying a prescribed organization by aggregate collaboration gains. These results change how attribution numbers in MAS papers should be read, which is why they carry decisive weight for me.

Critical weaknesses
  1. Generality is bounded by a single backbone and single-pass execution. All conclusions rest on one model (Qwen3.8-27B-FP8), and the runner executes every architecture — including the cyclic designs A4–A7 — in one deterministic pass, so the Debate architecture never actually debates and feedback edges never replay. The “execution protocol” whose fragility is measured is therefore a specific, somewhat degenerate family, yet the central claim (“contribution is not an intrinsic property of an agent or role”) is phrased at a fully general level. Whether the findings transfer to multi-round interaction, heterogeneous or stronger backbones, or genuinely iterative protocols is untested; this gap between evidence scope and claim scope is the main reason my recommendation stays at weak acceptance.
  2. Resolution limits and partially definitional evidence. Binary utility on 30-task subsets with K=8 coalition sampling produces sparse marginals — intermediate roles are exactly zero in 87–93% of runs, and the authors themselves show that the signs of near-zero roles are unstable under resampling — and the headline Chain–DAG sign reversal rests on 19 doubly-nonzero tasks (12 strict flips out of 150). Moreover, the A2/A3 final-answer contrast (0.000 → 0.499) follows almost by construction from the equation-2 validity gate: disabled runs have no selected evaluator-visible output and therefore zero utility by definition. The genuinely informative evidence is the A5 fallback analysis; presenting the definitional contrast as a headline effect risks overstating the empirical content.
Questions for authors
  1. Does the central finding replicate with a second backbone (e.g., a frontier closed model or a different open model)? Even a small-scale replication on one or two architectures would substantially affect my generality concern.
  2. How would the conclusions change under genuine multi-round execution of the cyclic architectures? Is the Aggregator/Finalizer dominance partly an artifact of never actually iterating (the Debate architecture never debates)?
  3. How much of the +0.499 final-answer effect is definitional (the eq. 2 validity gate forces zero utility when no authorized output exists) versus behavioral? Could the A5 fallback results alone carry the “authority ≠ control” claim?
  4. The aggregate sign reversal is driven by 12/150 tasks with opposite nonzero signs (19 tasks nonzero in both conditions). Is the reversal concentrated in particular benchmarks, and how should readers weight it?
  5. Given the 0.25 95th-percentile context–role error of the K=8 sampler in the A1 exact calibration, what sampling budget would be required for stable per-context attribution, and at what compute cost?
  6. Relative to Agents that Matter (Lu et al., 2026), which already established the sensitivity of removal-based attribution to coalition sampling, removal semantics, and target metrics, what is empirically new here beyond the intervention trio and the validity-gate framing? Please sharpen the delta.
5 comments
AuthorsOpenReview ↗

Response 1/6

  • Cross-backbone generality of the topology finding.

We conducted targeted cross-backbone experiments to examine whether the topology-dependent Verifier attribution observed with Qwen extends to other open models. In addition to the original Qwen model, we evaluated Gemma-4-31B-it-QAT and Llama-3.1-8B-Instruct under the controlled Chain and DAG conditions. For each backbone, we evaluate the complete system and Verifier null replacement on the same five benchmark subsets, with 30 matched tasks and three seeds per benchmark.

We define the topology effect as

δV=ΔV,DAGLOO−ΔV,ChainLOO\delta_V = \Delta_{V,\mathrm{DAG}}^{\mathrm{LOO}} - \Delta_{V,\mathrm{Chain}}^{\mathrm{LOO}}

where ΔVLOO\Delta_V^{\mathrm{LOO}} is the complete-system utility minus the utility under Verifier null replacement. Table 1 reports results averaged equally over the five benchmarks.

Table 1: Cross-backbone comparison of the Verifier topology effect, averaged equally over five benchmarks. UU denotes complete-system utility and ΔV\Delta_V denotes Verifier LOO attribution. The 95% confidence interval is computed using a stratified task-cluster bootstrap, retaining the three seeds and paired Chain–DAG conditions for each sampled task.

BackboneUChainU_{\mathrm{Chain}}UDAGU_{\mathrm{DAG}}ΔV,Chain\Delta_{V,\mathrm{Chain}}ΔV,DAG\Delta_{V,\mathrm{DAG}}δV\delta_V95% CI
Qwen0.4130.391+0.051+0.051−0.040-0.040−0.091-0.091[−0.144,−0.042][-0.144,-0.042]
Gemma0.4960.493+0.080+0.080+0.013+0.013−0.067-0.067[−0.113,−0.022][-0.113,-0.022]
Llama0.1890.184+0.024+0.024−0.009-0.009−0.033-0.033[−0.078,−0.005][-0.078,-0.005]

The direction of the aggregate topology effect is consistent across the three evaluated conditions: the Verifier's mean LOO decreases when moving from Chain to DAG. The decrease is 0.0910.091 with Qwen, 0.0670.067 with Gemma, and 0.0330.033 with Llama; all three corresponding bootstrap intervals exclude zero. The complete-system utility point estimates change only modestly in these aggregate comparisons; we do not interpret this observation as statistical equivalence between topologies.

The original aggregate mean sign reversal is not universal. Qwen's Verifier LOO changes from +0.051+0.051 to −0.040-0.040 and Llama's point estimates change from +0.024+0.024 to −0.009-0.009, whereas Gemma's decreases from +0.080+0.080 to +0.013+0.013 and remains positive. Thus, we interpret the results as cross-backbone support for the narrower observation that measured contribution can depend on the execution context, rather than as evidence that a particular sign reversal must occur for every backbone or benchmark.

At the benchmark level, the effect is strongest on AIME, where the DAG-minus-Chain change is −0.233-0.233 for Qwen and −0.189-0.189 for Gemma. Other benchmarks show smaller or more uncertain effects. This heterogeneity is consistent with our interpretation of contribution as conditional on the task and execution protocol.

Scope and limitations. This is a targeted validation of the Verifier topology contrast, not a replication of every role-prompt, permission, or coalition-attribution result in the paper. The Llama condition uses an adapted role-prompt template, omits the Qwen-specific /no_think marker, and uses a repetition penalty of 1.11.1 after development testing revealed repetitive outputs; this configuration is better suited to the model. Its comparison with Qwen and Gemma therefore changes both the backbone and aspects of the prompting and decoding setup.

The Llama complete-system utility is zero on ARC-AGI-2 and is 0.0110.011 under both topologies on AIME. These low success rates limit the resolution of LOO attribution and should not be interpreted as evidence that the Verifier has no contribution. We will report these limitations and avoid claiming that the observed sign or effect magnitude is universally preserved.

AuthorsOpenReview ↗

Thank you & Looking forward to further discussion

Dear Reviewer Sndg,

Thank you again for your careful review and for recognizing the experimental control and interpretive contribution of our work. We appreciate your thoughtful feedback. To facilitate further discussion, we briefly summarize our responses to your main concerns:

  • Generality across backbones and execution settings (Weakness 1; Questions 1–2): We evaluated the Verifier topology effect with Gemma and Llama and implemented a three-round A6 variant. The results support context-dependent attribution in the tested settings, while the sign reversal is not universal and Aggregator dominance partly reflects its terminal-node position.

  • Interpretation and resolution of the evidence (Weakness 2; Questions 3–5): We used shadow scoring and routing experiments to clarify the role of the validity gate, analyzed the sparsity and benchmark distribution of the sign reversal, and calibrated sampling budgets against different error tolerances.

  • Distinction from prior work (Question 6): We clarified how our controlled interventions and execution audits complement Agents that Matter, while acknowledging its prior findings on attribution sensitivity.

We understand that this is a busy time. If you have had an opportunity to review our responses, we would be grateful to hear whether they address your concerns or whether any points would benefit from further clarification. We would also appreciate your reconsideration of the evaluation if you find that our responses have addressed your concerns.

Thank you again for your time and consideration.

Best regards,
The Authors

AuthorsOpenReview ↗

Following Up Before the Discussion Deadline

Dear Reviewer Sndg,

Thank you again for your thoughtful review. As the discussion period is drawing to a close, we would greatly appreciate your feedback on whether our additional experiments and analyses have addressed your main concerns, particularly:

  • Whether the cross-backbone and three-round experiments sufficiently clarify the scope of our findings. (Weakness 1)
  • Whether our analyses of the sparse sign reversal, sampling error, and validity gate address your concerns about the interpretation of the evidence. (Weakness 2)

We have provided these results in our responses and plan to clarify their scope and limitations in the paper. Please let us know if any point would benefit from further clarification.

Thank you for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.