- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how to scale tool-using multi-agent LLM systems efficiently, comparing three choices: increasing each agent’s tool-use depth, adding agents, and adding communication rounds. It measures GPU energy directly across single-agent, independent, centralized, and decentralized systems. Its main finding is that increasing depth generally gives larger accuracy gains per unit of energy than adding agents, while additional communication rounds show no consistent average benefit when the total per-agent tool-step budget is held fixed.
Based on these results, the authors propose a configuration search policy that uses a single agent’s budget utilization to select a depth budget before deploying a multi-agent system. They evaluate the approach on FanOutQA, WorkBench, BrowseComp+, and SWE-bench Lite with Qwen3.5-9B, and repeat the scaling experiments on SWE-bench Lite with two additional models. The proposed policy reportedly reduces combined search and initial deployment energy by 51% relative to searching directly on the target multi-agent system, and by 59% relative to searching both depth and breadth, with accuracy losses of at most 0.5 percentage points on three of the four benchmarks.
Critical strengths
There are some critical stengths in the paper:
-
Direct energy measurements give the scaling study practical value. The paper measures GPU energy across depth, agent count, and communication rounds, rather than treating token counts as interchangeable units of cost. The supporting energy model is fitted on 17,250 task records and achieves an of approximately 0.961, with a median absolute percentage error of 7.6% under the measured serving setup. This gives the analysis a substantial empirical foundation. I find this consequential because practitioners need to know which scaling choices improve accuracy efficiently, and token-based comparisons can obscure the different costs of processing inputs and generating outputs.
-
The findings lead to a simple configuration policy with meaningful measured savings. Using a single agent’s budget utilization to select the depth of a multi-agent system is inexpensive and requires no task labels during configuration selection once the threshold is fixed. Table 1 reports an average 51% reduction in combined search and initial deployment energy relative to applying the same utilization rule directly on the target multi-agent topology, with an average accuracy difference of +0.4 percentage points. Compared with searching both depth and breadth, the reported saving is 59%, with accuracy losses no greater than 0.5 percentage points on three of four benchmarks. These results make the paper useful beyond its descriptive scaling analysis, although the benefit remains dependent on the benchmark and deployment setting.
Critical weaknesses
There are some critical weaknesses that I would like to mention in this paper:
-
The evaluation does not consistently measure successful task completion. On WorkBench, a task is counted as correct when the gold action set is contained in the union of all agents’ recorded calls, regardless of order or additional incorrect actions (Appendix B.3). This can reward broader exploration even when the resulting database state is wrong. BrowseComp+ similarly accepts a response containing the gold answer anywhere, including responses listing several competing candidates. These choices matter because increasing depth or agent count can increase both action coverage and candidate coverage without improving actual correctness. On SWE-bench Lite, “Independent” agents also share a worktree and run sequentially, so the measured breadth gain includes agents building on earlier agents’ edits. Although these choices are disclosed, stronger evaluation checks are needed before the results can support conclusions about task accuracy and independent ensembling.
-
The experiments support a narrower conclusion about communication than the paper’s explanation suggests. Fixing controls the maximum number of tool steps per agent, but increasing also changes execution segmentation, intermediate answers, review calls, and accumulated context. Without a comparison that keeps these factors fixed while removing or replacing peer messages, the experiment cannot isolate the contribution of communication content. Context limits further complicate the interpretation: Appendix C.1 reports that WorkBench Decentralized achieves 80–86% accuracy with 0–2% of tasks experiencing truncation at , compared with 56% accuracy and 57% truncation at . The authors acknowledge this limitation, but Figure 5 still includes settings up to . Moreover, the argument in Section 4.3 does not establish that another homogeneous agent’s message contains no useful additional evidence. The results therefore support limited average benefits under the tested protocol, rather than a general explanation that homogeneous communication is merely self-conditioning.
-
The configuration-search comparisons contain unresolved differences in coverage and evaluation protocol. Appendix E.1 states that the required breadth ladders were completed for depths selected at , including FanOutQA , and that Table 1 averages all three topologies over all 200 splits. However, the Table A16 caption states that FanOutQA Centralized has a full breadth ladder only at , while also reporting that 47% of depth probes select . These statements appear inconsistent and need clarification. Table A15 separately acknowledges that D retains only splits with complete breadth ladders, fewer than half the splits in several Centralized cells, and that its A and D rows are not paired. This does not establish that Table 1 is incorrect, but it makes the supporting comparisons difficult to audit. In addition, Section 5.1 selects using held-out accuracy from the same evaluation pool, without a separate final test of that threshold. These issues leave uncertainty about the reported policy advantage.
-
The depth-over-breadth conclusion is sensitive to the operating range, and its deployment scope remains insufficiently validated. Appendix C.2 explicitly reports that, when the depth fit is restricted to , BrowseComp+ has a topology-mean depth elasticity of 0.09, below its breadth elasticity of 0.19. Thus, the headline ordering partly depends on including the earliest depth increments and does not hold uniformly across benchmarks and budget ranges. Comparisons on the same task set and within comparable energy ranges would help establish when prioritizing depth is the better choice. The additional-model experiments cover only SWE-bench Lite (Section 4.5), while all actual agent runs use serial batch-1 serving (Appendix B.2). The batch-64 analysis re-prices existing trajectories using adjusted energy coefficients rather than evaluating agent execution under batching (Appendix A.3). These are useful checks, but they do not establish that the recommended scaling order remains preferable under realistic concurrent deployment.
Questions for authors
The following are my questions for the authors:
-
Do the main scaling results hold under stricter measures of task correctness? Appendix B.3 states that WorkBench accepts the gold action set within all recorded calls without penalizing additional incorrect actions, while BrowseComp+ accepts the gold answer appearing anywhere in the response. Have you checked whether the depth-versus-breadth ordering and policy comparisons remain consistent under WorkBench’s state-based evaluator and the official BrowseComp+ evaluation? Please also clarify whether SWE-bench accuracy uses the complete harness “resolved” criterion.
-
Can you reconcile the breadth coverage statements and clarify how D is evaluated? Appendix E.1 says the required FanOutQA ladders cover and Table 1 averages all 200 splits, whereas Table A16 says Centralized has a complete breadth ladder only at , despite 47% of depth probes selecting . What happens when a split selects a depth without a complete ladder? Please specify the included splits and any fallback rules for Table 1, and explain how these differ from the explicitly unpaired comparisons in Table A15.
-
What supports attributing the communication results to a lack of useful peer information? Fixing still changes execution segmentation, intermediate responses, and review calls. Is there a comparison that preserves these factors while removing or replacing peer messages? Separately, how does the distributional identity in Section 4.3 establish that a peer’s independently retrieved evidence cannot improve the receiving agent’s answer? Please distinguish the empirical finding under your protocol from the stronger theoretical interpretation, particularly given the truncation effects reported in Appendix C.1.
-
Was the utilization threshold evaluated independently of its selection? Section 5.1 selects from held-out accuracy across the evaluated settings. Were those same task outcomes subsequently used to report the policy’s final accuracy? Please clarify the separation between threshold selection and final evaluation, and whether the reported advantage persists with a threshold fixed before evaluating the test tasks. This would help establish how well the policy transfers beyond the tasks used to choose its setpoint.
These requests concern the validity and interpretation of the current results. Clear answers, with supporting evidence where available, could move my assessment toward weak acceptance.