- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper measures latency and energy for agentic design choices (multi-turn calls, RAG, orchestration, and parallelism) across 1,656 configurations, seven systems, and nine benchmarks, using Qwen3 and Gemma3 models up to 27B on A6000 Ada and L4 GPUs. For each choice it reports a pitfall and an opportunity: for example, model cascades beat repeated small-model calls at equal accuracy, retrieval often dominates RAG cost, orchestrated and parallel workflows leave GPUs idle, and configurations that are optimal at P50 often are not at P99.
Critical strengths
-
The end-to-end energy and latency of agent workflows is a timely and under-studied question, and measuring both across this many configurations is substantial work that others can build on.
-
Several findings are directly useful: RAG is often retrieval-bound, MagenticOne latency is dominated by the orchestrator, call share does not track latency share, and P50- and P99-optimal configurations differ. The mixed-hardware placement experiment is a concrete way to exploit this skew.
Critical weaknesses
-
The multi-turn cascades seem to stop using the ground-truth answer. Appendix B says that after each turn "the benchmark compares the agent's final answer with the reference answer; a correct answer ends the run," and Section 6 describes cascades as "terminating early on correct answers." If so, the reported accuracy is an oracle-assisted Pass@k, and the efficiency gain credited to the cascade comes from oracle stopping. Sections 3 and 6 rely on these configurations, so this needs to be clarified or rerun with a realistic stopping rule.
-
The iso-accuracy comparisons are fragile. GAIA L1 accuracies in Section 3 are multiples of 1/53, so the headline comparison is 8 correct answers against 8, with no repeated runs or confidence intervals, and it also pits a thinking model against non-thinking ones. The paper never describes how energy is measured or attributed, and some claims are overstated: the "severe" GPU under-utilization amounts to only 2% and 1.7% of total energy.
Post-rebuttal update The three-stage cascade without intermediate checks matched the repeated-call baseline’s reported accuracy (22.64%) at lower mean latency and energy. This partly addresses my concern about oracle stopping. The original Section 6 results still rely on ground-truth early stopping, and the four-stage no-check result falls outside the stated iso-accuracy threshold. Taking the new results into account, I’m raising my rating from 2 to 3.
Questions for authors
- Appendix B describes Multi-Turn AgentChat as stopping on a ground-truth check. Is the same oracle used by the Section 6 cascades and SelectorGroupChat, and how do the results change with a non-oracle stopping rule?
- How is energy measured (GPU, CPU, DRAM, idle) and attributed when GPUs are shared?
- Which configuration pair yields the 1,013× and 1,153× figures, what accuracy tolerance defines iso-accuracy, and how many runs is each point based on?
- Does the Section 3 conclusion hold when thinking mode is held fixed?
3 comments
Thank you again for your feedback! As the discussion period ends November 18, we would be grateful to know whether our responses address your concerns, and we are happy to clarify any remaining point.
Thanks for the additional experiments. The three-stage result without intermediate checks addresses part of my concern about oracle stopping, and I’ve raised my rating from 2 to 3.
Response to Reviewer UZyH
Thank you Reviewer UZyH for taking the time to carefully read and evaluate our paper and for providing thoughtful and constructive feedback. We appreciate your comments and suggestions, which have helped us identify areas where we can further strengthen our work. We address the main points of your review below:
Multi-Turn AgentChat Termination Criteria:
The Section 6 cascades use the same ground-truth stopping criterion as Multi-Turn AgentChat, while SelectorGroupChat, MagenticOne, and ReAct use non-oracle stopping rules. Without oracle-based stopping, cascades can still achieve iso-accuracy and outperform naive repeated calls, but the benefits are less pronounced because every query must traverse the full cascade, limiting the latency and energy savings from early stopping. We provide further details in Response to All Reviewers (A).
Hardware Profiling, Energy Evaluation.
We provide detailed description of our energy evaluation methodology in Official Comment C, and we will gladly include these details in a camera ready version of our paper.
Iso-Accuracy Evaluations.
For the GAIA L1 experiments which yielded the described variance, configurations are defined as iso-accuracy if the configuration is within 1% absolute task performance of the frontier with results measured over 3 trials per run.
Specifically, the configuration pair is the
gaia-level1-agentchat/1.7b-ntwith a single agent and 1 call compared againstgaia-level1-magenticone/1.7b-nt(4-agent MagenticOne); where latency ranges from 0.1s to 108.9 s (1,013.2x) and energy ranges from 25.7J to 29,673.8 J (1,152.6x). Additionally, we will release all of the configuration, code, and results for full clarity upon publication.For additional details on accuracy evaluations, see Response to All Reviewers (D)
Repeated Calls Across Isolated Non-Thinking and Thinking Structures:
The findings in Section 3 remain consistent when holding thinking mode fixed. For example, running a cascade of thinking models (0.6B -> 1.7B -> 8B -> 14B) is more efficient than repeatedly invoking a single larger 8B thinking model for the same number of turns to achieve the same accuracy level. We elaborate on this trade-off in more depth within Response to All Reviewers (B).