- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies active learning for communication-structure optimization in LLM-based multi-agent systems. The main motivation is that existing graph/communication optimization methods usually rely on randomly sampled training tasks, but different tasks are not equally informative for improving the communication structure. The paper empirically shows that communication-structure optimization is sensitive to the selected training tasks, and that simply increasing the number of randomly sampled tasks does not consistently improve downstream performance.
Critical strengths
-
A key strength is that the paper identifies a practically meaningful weakness in current communication-structure optimization pipelines: the final optimized graph can be highly sensitive to which training tasks are used.
-
The paper evaluates the method across several task formats, including QA, code generation, tool use, and embodied interaction, and also examines adversarial settings.
-
The paper does not focus only on mean performance, but also reports run-to-run variability, lower-tail metrics, and token cost.
Critical weaknesses
-
The main empirical concern is that many of the reported gains over random selection are fairly modest, while the statistical evidence is mixed. In the main benchmarks, improvements in mean accuracy are often small, especially in “benign” settings.
-
The proposed framework introduces substantial additional machinery, e.g., representative selection, EKI-based utility estimation, surrogate modeling, and batch selection. But the practical advantage over a simpler random-selection baseline is not always strong enough to justify this complexity.
-
The paper argues for an information-theoretic EKI-based criterion, but it remains unclear whether the reported gains truly come from the specific Bayesian/EKI formulation, or more simply from selecting a more diverse and filtered subset of tasks.
-
The paper does not convincingly show that the EKI-based utility is substantially more effective than simpler uncertainty-, diversity-, or heuristic-based strategies under matched computational budgets across the full benchmark suite.
-
Related work is quite insufficient and outdated.
Questions for authors
-
The paper reports improvements in mean performance, stability, and lower-tail metrics, but several bootstrap confidence intervals include zero. Could the authors clarify which main claims they consider statistically supported and provide significance analysis for all headline comparisons, not only selected cases in the appendix?
-
How much of the observed gain comes specifically from the EKI-based informativeness criterion, as opposed to the representative pre-selection stage or other pipeline components? Can the authors provide stronger controlled comparisons against simpler uncertainty/diversity-based selectors under matched total token budgets?
-
The paper emphasizes cost efficiency, but active selection itself introduces nontrivial overhead. Can the authors clarify under what practical budget regimes the proposed method is consistently preferable to simply training on more randomly sampled tasks?
If the authors can address my concerns well, I will be willing to increase the score.
4 comments
Response to Reviewer FFpD (2/2): Weaknesses 3-5 and Question 2-3
Weakness 3 and Question 2 (first part): ablation against diversity-based criteria
W3: It remains unclear whether the reported gains truly come from the specific Bayesian/EKI formulation, or more simply from selecting a more diverse and filtered subset of tasks.
Q2 (first part): How much of the observed gain comes specifically from the EKI-based informativeness criterion, as opposed to the representative pre-selection stage or other pipeline components?
We have an ablation study in Appendix D.6 that compares diversity only, informativeness only, and their combination. It shows that informativeness-only selection improves more than diversity-only selection. Nevertheless, our aim is to show that both are necessary for an efficient and effective method, not to compare them.
Weakness 4: informativeness vs. uncertainty-, diversity-, or other heuristic-based criteria
W4: The paper does not convincingly show that the EKI-based utility is substantially more effective than simpler uncertainty-, diversity-, or heuristic-based strategies under matched computational budgets across the full benchmark suite.
We do not argue that informativeness is better than other non-informativeness criteria. We focus on how to efficiently estimate information gain, so we compare different ways to estimate it (Section 5.4).
We simply chose a two-stage strategy, diversity first and then informativeness, as is common practice. Other combinations may perform better. We leave the search for the best combination to future work.
Question 2 (second part): cost-matched comparison
Q2 (second part): Can the authors provide stronger controlled comparisons against simpler uncertainty/diversity-based selectors under matched total token budgets?
The reviewer asks for a comparison against uncertainty- and diversity-based selectors under matched total token budgets.
Note that the diversity-based selector costs no tokens (it uses only the tasks themselves and does not run LLM rollouts), so we cannot conduct a matched-token comparison against the informativeness-based selector.
For the uncertainty-based selector, the reason is the same as in our reply to Weakness 4: this work does not claim that informativeness is better than uncertainty. Its focus is how to efficiently estimate information gain for task selection.
Question 3: cost efficiency
Q3: Can the authors clarify under what practical budget regimes the proposed method is consistently preferable to simply training on more randomly sampled tasks?
Table 1 (Section 5.2) shows that, under matched total cost, active learning achieves better downstream performance than training on randomly selected tasks.
Here, the total cost of active learning (additional selection cost + training cost) equals the total cost of random selection (training cost).
We have made the Table 1 easiler to understand.
Moreover, random selection does not reach active learning at any training budget. In Fig. 1, we test random selection from 0.05M to 3.54M training tokens, while the total cost of active learning is fixed at about 0.9M tokens. Random selection's mean accuracy first increases and then decreases. Even at its peak, it stays below the mean accuracy of active learning.
We interpret the question as asking for the break-even point, where both methods reach the same performance at the same total cost. This point answers when to use active learning, while our work answers how to perform active learning. Finding the break-even point is an interesting problem, and we leave it for future work. We now state it as a limitation in Section 6 (lines 507–509) and leave it for future work.
Weakness 5: related work
W5: Related work is quite insufficient and outdated.
We expanded Section 3 with recent work on data selection for LLMs (Xia et al., 2024; Zhang et al., 2025c; Lin et al., 2025; Chen et al., 2025) and recent information-theoretic criteria (Guilhoto et al., 2026; Durmazkeser et al., 2026; Huang et al., 2026) (lines 126–129). We would also welcome any specific references the reviewer has in mind.
For communication-structure optimization, the introduction cites the latest methods published before May 2026. We also checked for newer work and added Xu et al. (2026) in Section 3 (line 121).
We hope these responses address the reviewer’s concerns. If so, we would be grateful if the reviewer would consider raising the score.
Thanks for the author's response. Here is my opinion:
Q1 actually confirmed rather than disproved W1. It indicates that the significant positive results were mainly confined to the "attack" scenario and the "InterCode-SQL" new benchmark, while there was no significant result in the "benign" scenario.
W5 has responded appropriately.
W2 The author states that EKI is the core component, and the other components are merely for budget savings and can be removed (Appendix D.6), yet the selection of tasks can still be done with EKI, only at the cost of more tokens. This is a reasonable narrowing defense, but it still fails to provide a direct argument regarding the cost-effectiveness of this complex approach compared to "simply using more random tasks".
Q3 the author bypassed the core of my question - "In what budget range is this method more stable and effective?" - instead of providing an answer.
Overall, some responses (Q1, W5) were honest and thorough, while Q3 and W2 made substantive progress; however, regarding your core methodological doubts (W3, W4, and Q2), the replies were evasive by "redefining the scope of the claim" rather than providing answers. Therefore, from the perspective of "whether they accurately answered your questions": The claims were somewhat comprehensively and honestly narrowed, but the two most critical empirical gaps were not addressed. If based only on this reply, I don't think it is sufficient to raise the score further from "Weak rejection". My current highest rating is "Weak rejection".
Response to Follow-up Comments
We thank the reviewer for raising your score and for the follow-up comments. Our responses are below.
Q1 / W1: Benign scenario
Reviewer: Q1 actually confirmed rather than disproved W1. It indicates that the significant positive results were mainly confined to the "attack" scenario and the "InterCode-SQL" new benchmark, while there was no significant result in the "benign" scenario.
Response. We agree with this observation. We believe the main reason is that the benign scenario leaves little room for improvement, as explained in Appendix D.3.1.
In the revision, we propose to make the following changes:
- We will narrow the main claim of the paper and focus it on the attack scenario.
- We will move the benign results to the appendix.
Do they address your concern?
W5
Reviewer: W5 has responded appropriately.
Response. Thank you for confirming.
W2: Cost-effectiveness compared with more random tasks
Reviewer: The author states that EKI is the core component, and the other components are merely for budget savings and can be removed (Appendix D.6), yet the selection of tasks can still be done with EKI, only at the cost of more tokens. This is a reasonable narrowing defense, but it still fails to provide a direct argument regarding the cost-effectiveness of this complex approach compared to "simply using more random tasks".
Response. We understand the question as: under the same total cost, does our approach perform better than simply using more random tasks?
Yes, Fig. 1 and Table 1 answer this question. Under matched total token cost, our approach outperforms random task selection on every metric for both benchmarks:
| Benchmark | Metric | Ours | Random | Δ |
|---|---|---|---|---|
| MMLU | Mean | 78.32% | 77.35% | +0.97 pp |
| Q1 | 76.47% | 75.82% | +0.65 pp | |
| Worst-25% | 76.25% | 73.85% | +2.40 pp | |
| GSM8K | Mean | 93.83% | 92.64% | +1.19 pp |
| Q1 | 93.15% | 92.11% | +1.04 pp | |
| Worst-25% | 92.93% | 91.73% | +1.20 pp |
Since the token cost is the same, we believe this is a direct evidence that our approach is more cost-effective than using more random tasks.
Q3: Across budget ranges
Reviewer: the author bypassed the core of my question - "In what budget range is this method more stable and effective?" - instead of providing an answer.
Response. We now understand the question as: when the total budget changes, does our approach still outperform cost-matched random selection? Yes.
We run the cost-matched test on more budget levels. We did not have time to test a wider and denser range of budgets, since we received this clarification less than one day ago. We were able to add a few budget levels, which took several hours to run. We report results on MMLU (benign), since the reviewer's concern is mainly about the improvement in this setting:
| Total cost (M tokens) | Ours vs. Random | Runs |
|---|---|---|
| ~1.2 | 81.96% vs 81.37% (↑) | 10 vs 6 |
| ~2.2 | 81.45% vs 80.70% (↑) | 10 vs 6 |
| ~4.3 | 82.29% vs 81.48% (↑) | 10 vs 3 |
At each matched cost, our approach achieves a higher mean accuracy than random selection, by 0.59 pp, 0.75 pp, and 0.81 pp at about 1.2M, 2.2M, and 4.3M tokens, respectively. We don't have time to revise the paper. We will include this results later.
Comparison with cost-matched uncertainty-based sampling. In case the question also concerns how our information-gain criterion compares with uncertainty sampling, we ran a new comparison against cost-matched uncertainty-based sampling at three cost levels on MMLU benign (each for 10 runs).
| Cost (M tokens) | Method | Mean | Q1 | Worst-25% |
|---|---|---|---|---|
| level 1 (~1.2) | Ours | 81.96% (↑) | 81.05% (↑) | 80.78% (↑) |
| Uncertainty | 80.83% | 79.74% | 79.74% | |
| level 2 (~2.2) | Ours | 81.45% (X) | 80.39% (↑) | 78.98% (X) |
| Uncertainty | 81.70% | 79.74% | 79.30% | |
| level 3 (~4.3) | Ours | 82.29% (↑) | 81.86% (↑) | 81.18% (↑) |
| Uncertainty | 80.39% | 79.08% | 78.65% |
At cost levels 1 and 3, our approach outperforms uncertainty sampling by at least 1 pp on all three metrics. At cost level 2, uncertainty sampling is slightly better on Mean and Worst-25% (by 0.25 and 0.32 pp), while our approach is better on Q1 (by 0.65 pp).
This comparison is not the focus of our work, and we do not claim that either criterion is better in general.
We hope our responses address your two main concerns:
- the modest improvement in the benign scenario;
- the comparison with cost-matched random selection at different budgets.
If so, we would be grateful if you would consider raising your score further.
Response to Reviewer FFpD (1/2): Weaknesses 1-2 and Question 1
We thank the reviewer for the detailed comments and for being open to raising the score. We will revise the paper accordingly.
Weakness 1 and Question 1: size of the gains and statistical support
Claims and support:
The table below gives the analysis for all headline comparisons of active against random selection. The lower-tail tests on InterCode-SQL and the intervals on HumanEval are new. ALFWorld has only four to five runs per method. The smallest attainable p-value of a paired permutation test is then 0.0625, which cannot be significant at the 0.05 level, so we do not test significance on ALFWorld. We added this table to Appendix D.3 (Table 8) and stated there which claims are statistically supported (lines 1689–1692). Section 5.2 refers to these tests (line 389).
Why some gains are smaller:
For some benchmarks (typically the easier ones), there is not much room for improvement. On these benchmarks, graph optimization with randomly selected tasks improves performance by only about 1 pp, similar to Zhang et al. (2025b). Active selection can at most reach what graph optimization can achieve. Its gain over random selection comes from avoiding the poor task sets that lead to failure cases. This lifts the lower tail and, with it, the mean. So the mean gains on these benchmarks are expected to be small.
However, on the new benchmarks that we add and the original work (Zhang et al., 2025b) does not use, i.e., InterCode-SQL and ALFWorld, the improvements are relatively large (typically 3–8 pp).
We added this explanation as Appendix D.3.1.
Weakness 2: complexity
The key feature of our method is to use EKI to approximate information gain for task selection. All other components are used to save computational budget, as stated at the beginning of Section 4.5. Readers can remove the other components: EKI-based information gain alone can still select good tasks (Appendix D.6), only at the cost of more selection tokens. We added this point to Section 4.5 (lines 331–332).