- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies centralized and decentralized LLM multi-agent coordination through a configuration index controlling decomposition granularity and per-unit routing probability. Using MetaAgent and AgentNet with a shared model and five-agent roster, it evaluates 15 configurations across five benchmark collections. It reports task-dependent accuracy peaks, changes in agent-allocation profiles, and generally lower run-to-run variance under atomic delegation.
Critical strengths
The systematic sweep across several task domains provides useful descriptive evidence about coordination choices. Shared model and agent configurations, repeated runs, and matched-routing comparisons improve interpretability. The paper also appropriately acknowledges that intermediate accuracy peaks do not establish benefits from mixing branches.
Critical weaknesses
The comparisons leave routing effects difficult to distinguish from decomposition and execution differences. Intermediate configurations introduce Coordinator-level decomposition and aggregation that the endpoints bypass. For example, BBH accuracy increases from 0.217 at d=0 to 0.750 at d=0.0001, where routing remains almost entirely centralized. Appendix B.3 specifies different branch temperatures and execution limits, without reporting total token or model-call costs. Section 5.3 acknowledges these confounds, but they limit the design guidance supported by the descriptive findings.
The evidence for reliable configuration preferences and stability differences is limited. Evaluation subsets contain 90–168 items, with only two or three runs per condition. Table 1 reports observed ranges, without uncertainty estimates for paired accuracy differences or variance ratios. The highest observed mean among 15 configurations does not establish a reproducible preference. Appendix C.2 explains condition inclusion, but the variance aggregation remains unclear. One model and one pair of branch implementations further limit generality. These limitations motivate my weak-reject recommendation.
Questions for authors
- Which routing conclusions hold within a fixed granularity regime? A focused comparison with identical decompositions and aggregation, matched decoding, and reported computation would help clarify the performance differences.
- Can paired uncertainty analyses of existing task-level outcomes establish how reliably the preferred configurations outperform alternatives?
- How are per-condition variances aggregated and weighted, and how sensitive are the ratios to individual high-variance conditions?
I would consider raising my rating if additional analysis, focused controls, or appropriately narrowed claims make the performance and stability findings sufficiently reliable and informative.
4 comments
[W2, Q2] What do the paired uncertainty analyses establish?
They quantify task-level uncertainty for the selected configuration contrasts. They do not establish that the highest observed configuration is a reproducible global optimum.
For each task , correctness is averaged across the observed runs at configuration , giving . We then calculate the paired difference
The task-level bootstrap resamples task identities while preserving the paired outcomes for each selected comparison. It therefore estimates uncertainty over tasks conditional on the observed runs. It does not replace additional independent runs or quantify all variability expected from future reruns.
The following table reports the selected intermediate max–min comparisons.
| Setting | , accuracy | , accuracy | Bootstrap 95% CI | Nominal -value |
|---|---|---|---|---|
| BBH | 0.9999, 0.887 | 0.5001, 0.747 | [0.078, 0.204] | |
| AIME | 0.6, 0.667 | 0.9, 0.563 | [0.048, 0.163] | |
| MMLU-Pro, BBH-trained | 0.5, 0.856 | 0.6, 0.818 | [0.012, 0.065] | 0.00570 |
| MMLU-Pro-trained | 0.25, 0.848 | 0.9999, 0.586 | [0.211, 0.312] | |
| R-Bench-T | 0.5001, 0.730 | 0.0001, 0.633 | [0.048, 0.149] | |
| SMFR | 0.0001, 0.411 | 0.9999, 0.295 | [0.045, 0.191] | 0.00168 |
We also distinguish the comparisons’ purposes. The intermediate max–min contrasts describe the observed sweep range. The endpoint contrasts examine decomposition-associated workflow changes. The within-granularity contrasts address routing sensitivity. Some intervals exclude zero; others do not. An interval containing zero leaves that contrast unresolved and is not evidence of equivalence.
The reported -values are nominal and unadjusted for the number of comparisons examined. The max–min configurations were selected after inspecting the sweep, so their pointwise intervals and -values should not be interpreted as selection-adjusted evidence for the best configuration. They answer the narrower question of how the selected observed outcomes differ.
Appendix G.2 reports these comparisons separately. The revised accuracy claim concerns promising observed intermediate configurations, while the proposed validation procedure separates configuration selection from held-out evaluation.
[W2, Q3] How are condition variances aggregated and weighted?
We first calculate the sample variance of accuracy across runs separately for each complete DoF condition:
\left(\mathrm{Acc}_{d,r}-\overline{\mathrm{Acc}}_d\right)^2.$$ We then average these condition variances with equal weight within each granularity regime: $$\mathrm{Var}_g=\frac{1}{\lvert\mathcal{D}_g\rvert}\sum_{d\in\mathcal{D}_g}s_d^2, \qquad \rho=\frac{\mathrm{Var}_{\mathrm{subtask}}}{\mathrm{Var}_{\mathrm{atomic}}}.$$ Thus, the statistic averages run-to-run variation within conditions; it does not pool accuracy values across different DoF settings. Each included condition has equal weight, and each regime is normalized by its own number of included conditions. The endpoints $d=0,1$ and conditions without their complete planned run cohort are excluded. The planned counts are three runs for each BBH-trained setting and two for the MMLU-Pro-trained setting. **How sensitive is the ratio to individual high-variance conditions?** We added symmetric top-1 trimming: remove the highest-variance condition from each regime, recompute both arithmetic means over the remaining conditions, and calculate their ratio. | Setting | Untrimmed ratio | Symmetric top-1 trimmed ratio | |---|---:|---:| | BBH | 1.108 | 5.074 | | AIME | 1.854 | 1.958 | | MMLU-Pro, BBH-trained | 3.183 | 2.620 | | MMLU-Pro-trained | 1.174 | 0.726 | | R-Bench-T | 2.432 | 1.937 | | SMFR | 4.012 | 1.497 | All six untrimmed ratios exceed one. Five remain above one after symmetric trimming; MMLU-Pro-trained falls below one. The lower atomic variance is therefore a recurring pattern in the evaluated settings, but its robustness differs by setting. This trimming analysis measures dependence on extreme conditions; it is not a confidence interval for the variance ratio. The paired accuracy intervals also do not provide uncertainty intervals for variance ratios. We therefore report the stability finding through the observed run variances and the specified sensitivity check. We have incorporated the trimmed results into Table 1, the full calculation and cohort rules into Appendix C.2, and the resulting conclusion into Section 5.3.Thanks for the detailed reply
Thanks for the detailed reply! I've updated my rating upwards.
Thank You for Your Feedback and Reassessment
Thank you for your feedback and for updating your rating! We greatly appreciate your recognition that the additional analyses strengthen the evidence for task-dependent coordination choices and the reported stability patterns, and address the main concerns.
Response to Reviewer boPo
Thank you for the specific requests about routing controls, uncertainty, and variance sensitivity. We address the scope of each analysis directly below.
[W1, Q1] Which routing conclusions hold within fixed granularity, and what remains uncontrolled?
The supported conclusion is that workflow performance varies with routing within a granularity regime. It is not that mixed routing consistently outperforms decomposed single-branch execution.
We separate this conclusion from endpoint-to-intermediate changes. Those comparisons introduce Coordinator decomposition and aggregation and therefore cannot isolate branch mixing. The BBH example highlighted in the review is one such comparison. The following selected endpoint contrasts illustrate the associated workflow changes.
Within a granularity regime, the requested unit scope, decomposition prompt contract, and aggregation strategy remain fixed while routing varies. The selected paired comparisons below identify routing-dependent differences in several settings; other comparisons remain unresolved within their uncertainty intervals.
We did not cache and replay identical decompositions. The recovered R-Bench-T example at d=0.2 and d=0.4 contains 15 and 14 units. Both follow five broad stages, but differ locally. It supports comparable high-level decomposition structure rather than exact identity. The aggregate decomposition-scale summaries are:
The decoding and execution settings also differ between branches. Coordinator and MetaAgent calls use temperature 0.3; AgentNet uses temperature 0 and top-p=1.0. MetaAgent permits twelve planning rounds and one redo per step; AgentNet permits three forward hops and five executions per task. These settings are held fixed within each branch, not matched across branches.
Likewise, the shared 2,048-token output limit is per call, not a matched total-task budget. We have not established a cost-controlled routing advantage. The results concern assignment between these implemented workflows, including their execution behavior. Appendix B.3 gives the settings. We have incorporated the decomposition evidence into Appendix G.1 and the selected paired comparisons into Appendix G.2. Section 5.3 now distinguishes these findings from gains associated with the decomposed workflow.