- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes E2-Explainer, a post-hoc, model-agnostic framework to identify compact, task-preserving communication subgraphs in LLM-based multi-agent systems (MAS). It estimates a Granger-style causal contribution for each edge by masking channels and measuring changes in task score and result-level semantic entropy, then projects high-utility edges into executable, budgeted subgraphs and distills these into an amortized explainer that predicts subgraphs at test time without repeated edge evaluations. Experiments across six benchmarks and four topology generators report 20–25% token reductions with maintained or improved accuracy, plus transfer across unseen topology generators and agent scales.
Critical strengths
-
Using causal technologies to explain MAS topologies sounds interesting.
-
The evaluation is across various benchmarks.
-
Offers a framework that can be attached to multiple existing topology designers with applicability.
Critical weaknesses
1. Limited evaluation. Firstly, the baseline comparisons are limited. Although the paper discusses AgentPrune [1], it does not include AgentDropout [2]. Moreover, for general-purpose topology designers (e.g., G-Designer [3]), edge-wise normalization can be directly into the optimization objective for sparse topologies. Secondly, the evaluation uses only Qwen3-8B as the LLM backbone, without experiments on other widely used open-source and closed-source LLMs.
2. Less convincing motivation. As illustrated in Figure 1 and discussed in the Introduction, the learned topologies may contain redundant edges and communication patterns that are merely correlated with task success rather than causally contributing to it. However, this issue has already been extensively discussed in AgentPrune [1] and AgentDropout [2]. The paper should therefore clarify why existing MAS pruning methods still retain redundant edges or remove useful ones, and explain how the proposed method addresses these limitations.
3. Unclear presentation. For example, in Table 1, the Method column spans two lines, whereas the Metric column spans three, and the token consumption values are not directly reported. In addition, Tables 2, 3, and 4 are stacked together, making the results less readable.
4. Lack of code availability. The authors do not provide a code repository, making it difficult to reproduce the experiments and assess the reliability of the reported results.
References
[1] CUT THE CRAP: AN ECONOMICAL COMMUNICATION PIPELINE FOR LLM-BASED MULTI-AGENT SYSTEMS (ICLR 2025).
[2] AgentDropout: Dynamic Agent Elimination for Token-Efficient and High-Performance LLM-Based Multi-Agent Collaboration (ACL 2025).
[3] G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks (ICML 2024).
Questions for authors
-
Comparison with RL-based pruning methods. Why does E2-Explainer outperform RL-based pruning methods such as AgentPrune and AgentDropout? Could the authors compare RL-based pruning with causal analysis in terms of effectiveness, scalability, and resource consumption? In particular, I would expect RL-based methods to scale better than the edge-wise causal analysis used by E2-Explainer.
-
Additional baseline. Could the authors include a comparison with AgentDropout?
-
More LLM backbones. Could the authors evaluate E2-Explainer on additional open-source and closed-source LLM backbones to demonstrate its generalizability?
-
Code availability. Could the authors release the code through an anonymous GitHub repository to facilitate reproducibility?
17 comments
W1 & Q3: Generalizability across LLM backbones
To evaluate the generalizability of E2-Explainer across different LLM backbones, we conducted additional experiments with two representative models: an open-source model (Llama-3.1-8B) and a closed-source model (GPT-4o-mini), in addition to the original Qwen3-8B backbone. We evaluated all three backbones on MMLU and HumanEval.
As shown in the results below and in Table 6 of the revised manuscript, E2-Explainer consistently improves task accuracy while reducing online token consumption across all three backbones. These results further demonstrate that E2-Explainer is not tied to a specific LLM backbone and can generalize effectively across different model architectures.
| Backbone | Benchmark | Accuracy: Original → +E2-Explainer (%) | Online token reduction |
|---|---|---|---|
| Qwen3-8B | MMLU | 78.65 → 79.74 | 31.5% |
| Qwen3-8B | HumanEval | 88.40 → 90.69 | 19.3% |
| Llama-3.1-8B | MMLU | 72.55 → 75.16 | 26.4% |
| Llama-3.1-8B | HumanEval | 62.50 → 65.00 | 11.0% |
| GPT-4o-mini | MMLU | 80.39 → 82.35 | 20.7% |
| GPT-4o-mini | HumanEval | 85.00 → 86.25 | 10.8% |
W2: Why does redundancy remain after AgentPrune/AgentDropout?
Communication redundancy is a well-recognized challenge in LLM-based MAS, and existing methods such as AgentPrune and AgentDropout have proposed various pruning strategies to address it. However, our key observation is that substantial communication redundancy can persist even after these efforts.
As demonstrated in our original motivation experiment (Figure 1), G-Designer-generated topologies still contain substantial removable communication despite incorporating sparsity-promoting regularization. Similarly, our original Appendix Figure 4 reveals residual redundancy in OFA-MAS-generated topologies. To further address the reviewer's concern, we have extended the MMLU random edge-masking analysis to AgentPrune and AgentDropout, with the results reported in Appendix Figure 5 of the revised manuscript. Together with the improvements achieved by E2-Explainer on both methods (Table 1), these results further demonstrate that substantial redundancy can remain even in topologies optimized by existing pruning methods.
Regarding why existing MAS pruning methods may still retain redundant edges or remove useful ones, we have clarified this limitation in the Introduction (Lines 78–84) in the original version. Specifically, existing methods primarily optimize communication topologies through aggregate task-level rewards or sparsity objectives, without explicitly attributing task outcomes to individual communication decisions. Consequently, they may fail to distinguish redundant communication from critical task-preserving substructures.
To address this limitation, E2-Explainer explicitly investigate the causal contribution of individual communication edges to the task outcomes achieved by the overall communication topology. Based on these intervention-derived signals, we identify critical communication paths and construct compact, executable subgraphs that preserve task-relevant information flow. We further distill these subgraphs into an amortized explainer, enabling efficient post-hoc refinement of topologies produced by different optimization methods without repeated edge-level interventions at inference time. The aforementioned discussion can also be found in the Introduction (Lines 86–107) and Method sections of the original manuscript.
W3: Unclear presentation.
We thank the reviewer for pointing out these presentation issues. We have carefully revised the manuscript to improve the clarity and readability of the experimental results. Specifically, we have (1) corrected the inconsistent row spans between the Method and Metric columns in Table 1, (2) reorganized Tables 2–4 to improve their layout and readability, and (3) explicitly reported the previously missing token consumption values in Appendix Table 9 of the revised manuscript.
For the reviewer's convenience, we also provide the token consumption results in the table below, with all values reported in thousands of tokens (K):
| Method | Variant | MMLU | GSM8K | MultiArith | SVAMP | AQuA | HumanEval | Total |
|---|---|---|---|---|---|---|---|---|
| ARG-Designer | Original | 626 | 2,699 | 1,678 | 4,387 | 1,067 | 463 | 10,919 |
| + E2-Explainer | 550 | 2,281 | 1,287 | 3,461 | 749 | 396 | 8,724 | |
| OFA-MAS | Original | 495 | 7,637 | 1,445 | 2,302 | 690 | 2,207 | 14,777 |
| + E2-Explainer | 421 | 5,560 | 1,076 | 1,754 | 567 | 1,616 | 10,994 | |
| AgentPrune | Original | 1,430 | 6,782 | 2,660 | 5,504 | 1,298 | 709 | 18,384 |
| + E2-Explainer | 1,220 | 4,971 | 1,934 | 4,244 | 804 | 634 | 13,806 | |
| G-Designer | Original | 985 | 1,549 | 3,069 | 11,378 | 1,340 | 726 | 19,047 |
| + E2-Explainer | 675 | 867 | 2,208 | 9,092 | 838 | 586 | 14,266 | |
| AgentDropout | Original | 2,354 | 13,764 | 5,474 | 9,650 | 2,652 | 423 | 34,317 |
| + E2-Explainer | 1,800 | 11,316 | 4,269 | 7,341 | 1,709 | 363 | 26,800 |
All values represent the total number of prompt and completion tokens consumed over the corresponding evaluation dataset.
W4 & Q4: Code availability
We have provided the anonymized source code and evaluation scripts in the supplementary material. These materials include the core implementation and the main training and evaluation pipelines, enabling reviewers to examine the implementation details and reproduce the reported experimental results.
We thank the reviewer for the constructive feedback. We have addressed the concerns below and made corresponding revisions to the manuscript. All revisions are highlighted in blue in the revised PDF.
W1 & Q2: Limited baselines
Comparison with AgentDropout:First, following the reviewer's suggestion, we have included AgentDropout as an additional baseline in our main comparison. We adopt the official implementation of AgentDropout, with the only modification being the use of two communication rounds instead of the four rounds reported in the original paper for HumanEval, due to time constraints. All other implementation details and hyperparameters remain identical to those in the official implementation.
As shown in the table below, E2-Explainer can further identify critical task-preserving substructures within AgentDropout-optimized topologies, leading to improvements in both performance and efficiency. Specifically, E2-Explainer improves average accuracy by 1.27 percentage points while reducing online token consumption by 21.9%. These results further demonstrate the generality and plug-and-play capability of E2-Explainer, which can complement existing communication topology optimization methods by identifying essential communication subgraphs to further improve task performance and communication efficiency.
Detailed results are also provided in Table 1 and Table 9 of the Appendix in the revised manuscript.
Comparison with edge-wise normalization:Furthermore, we would like to clarify that G-Designer, which is included in our main comparison, already incorporates a regularization mechanism similar to the edge-wise normalization suggested by the reviewer. Specifically, G-Designer employs nuclear-norm regularization on a low-dimensional matrix to encourage low-rank structure and promote sparse communication topologies, referred to as the "regularization objective" in the original paper of G-Designer. However, both our motivation experiments in Figure 1 and the comparison with G-Designer in Table 1 demonstrate that such regularization alone is insufficient to effectively eliminate redundant communication dependencies. In contrast, E2-Explainer can further identify and remove redundant communication structures, leading to additional improvements in both task performance and communication efficiency.
Overall, we have conducted extensive comparisons to demonstrate the effectiveness and plug-and-play capability of E2-Explainer, including comparisons involving AgentDropout and the edge-wise regularization mechanism highlighted by the reviewer. Specifically, as shown in Table 1, we integrate E2-Explainer with five representative state-of-the-art topology optimization methods (ARG-Designer, OFA-MAS, AgentPrune, AgentDropout, and G-Designer). These results consistently demonstrate that E2-Explainer serves as a model-agnostic, post-hoc framework that identifies compact, task-critical causal subgraphs from communication topologies produced by diverse topology designers, further improving both performance and efficiency. The experimental evaluation is comprehensive and thorough.