acceptodds
Under review as a conference paper at ICLR 2027

Performance and Energy Implications of Designing Efficient Agentic Workflows: Pitfalls and Opportunities

Abstract

Agentic systems extend traditional LLM inference by coordinating specialized agents with external tools and data sources to execute increasingly complex, multi-step tasks. However, translating these capabilities into cost-effective real-world deployments remains an open challenge. Agentic design choices (including call frequency, tool integration, and orchestration) can substantially change latency and energy while maintaining similar accuracy. In our evaluation, iso-accurate configurations differ by as much as 238s and 1,013x in latency, or 177.8kJ and 1,153x in energy per task. In this paper, we construct 1,656 agentic systems across a diverse set of challenging, state-of-the-art agentic benchmarks and characterize the performance and energy implications of key agentic design decisions from a systems perspective. Through this characterization, we identify opportunities to improve the efficiency of agentic systems that can be used to guide more efficient deployments.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Agentic Systems, Energy Efficiency

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_UZyHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper measures latency and energy for agentic design choices (multi-turn calls, RAG, orchestration, and parallelism) across 1,656 configurations, seven systems, and nine benchmarks, using Qwen3 and Gemma3 models up to 27B on A6000 Ada and L4 GPUs. For each choice it reports a pitfall and an opportunity: for example, model cascades beat repeated small-model calls at equal accuracy, retrieval often dominates RAG cost, orchestrated and parallel workflows leave GPUs idle, and configurations that are optimal at P50 often are not at P99.

Critical strengths
  1. The end-to-end energy and latency of agent workflows is a timely and under-studied question, and measuring both across this many configurations is substantial work that others can build on.

  2. Several findings are directly useful: RAG is often retrieval-bound, MagenticOne latency is dominated by the orchestrator, call share does not track latency share, and P50- and P99-optimal configurations differ. The mixed-hardware placement experiment is a concrete way to exploit this skew.

Critical weaknesses
  1. The multi-turn cascades seem to stop using the ground-truth answer. Appendix B says that after each turn "the benchmark compares the agent's final answer with the reference answer; a correct answer ends the run," and Section 6 describes cascades as "terminating early on correct answers." If so, the reported accuracy is an oracle-assisted Pass@k, and the efficiency gain credited to the cascade comes from oracle stopping. Sections 3 and 6 rely on these configurations, so this needs to be clarified or rerun with a realistic stopping rule.

  2. The iso-accuracy comparisons are fragile. GAIA L1 accuracies in Section 3 are multiples of 1/53, so the headline comparison is 8 correct answers against 8, with no repeated runs or confidence intervals, and it also pits a thinking model against non-thinking ones. The paper never describes how energy is measured or attributed, and some claims are overstated: the "severe" GPU under-utilization amounts to only 2% and 1.7% of total energy.


Post-rebuttal update The three-stage cascade without intermediate checks matched the repeated-call baseline’s reported accuracy (22.64%) at lower mean latency and energy. This partly addresses my concern about oracle stopping. The original Section 6 results still rely on ground-truth early stopping, and the four-stage no-check result falls outside the stated iso-accuracy threshold. Taking the new results into account, I’m raising my rating from 2 to 3.

Questions for authors
  1. Appendix B describes Multi-Turn AgentChat as stopping on a ground-truth check. Is the same oracle used by the Section 6 cascades and SelectorGroupChat, and how do the results change with a non-oracle stopping rule?
  2. How is energy measured (GPU, CPU, DRAM, idle) and attributed when GPUs are shared?
  3. Which configuration pair yields the 1,013× and 1,153× figures, what accuracy tolerance defines iso-accuracy, and how many runs is each point based on?
  4. Does the Section 3 conclusion hold when thinking mode is held fixed?
3 comments
AuthorsOpenReview ↗

Response to Reviewer UZyH

Thank you Reviewer UZyH for taking the time to carefully read and evaluate our paper and for providing thoughtful and constructive feedback. We appreciate your comments and suggestions, which have helped us identify areas where we can further strengthen our work. We address the main points of your review below:

Multi-Turn AgentChat Termination Criteria:

The Section 6 cascades use the same ground-truth stopping criterion as Multi-Turn AgentChat, while SelectorGroupChat, MagenticOne, and ReAct use non-oracle stopping rules. Without oracle-based stopping, cascades can still achieve iso-accuracy and outperform naive repeated calls, but the benefits are less pronounced because every query must traverse the full cascade, limiting the latency and energy savings from early stopping. We provide further details in Response to All Reviewers (A).

Hardware Profiling, Energy Evaluation.

We provide detailed description of our energy evaluation methodology in Official Comment C, and we will gladly include these details in a camera ready version of our paper.

Iso-Accuracy Evaluations.

For the GAIA L1 experiments which yielded the described variance, configurations are defined as iso-accuracy if the configuration is within 1% absolute task performance of the frontier with results measured over 3 trials per run.

Specifically, the configuration pair is the gaia-level1-agentchat/1.7b-nt with a single agent and 1 call compared against gaia-level1-magenticone/1.7b-nt (4-agent MagenticOne); where latency ranges from 0.1s to 108.9 s (1,013.2x) and energy ranges from 25.7J to 29,673.8 J (1,152.6x). Additionally, we will release all of the configuration, code, and results for full clarity upon publication.

For additional details on accuracy evaluations, see Response to All Reviewers (D)

Repeated Calls Across Isolated Non-Thinking and Thinking Structures:

The findings in Section 3 remain consistent when holding thinking mode fixed. For example, running a cascade of thinking models (0.6B -> 1.7B -> 8B -> 14B) is more efficient than repeatedly invoking a single larger 8B thinking model for the same number of turns to achieve the same accuracy level. We elaborate on this trade-off in more depth within Response to All Reviewers (B).

Thank you again for your feedback! As the discussion period ends November 18, we would be grateful to know whether our responses address your concerns, and we are happy to clarify any remaining point.

Thanks for the additional experiments. The three-stage result without intermediate checks addresses part of my concern about oracle stopping, and I’ve raised my rating from 2 to 3.

Reviewer_bLrwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper measures how agentic LLM workflow design affects accuracy, latency, and energy use. The authors explore 1,656 configurations across seven agentic system designs, including multi-turn model calls, ReAct, retrieval-augmented generation, MagenticOne, selector-based group chat, multi-agent debate, and parallel MagenticOne. The evaluation spans nine benchmark/level combinations, multiple model sizes and families, and two GPU platforms, with additional CPU-side retrieval experiments. The focus is on end-to-end systems: workflow structure, tool settings, orchestration, parallelism, and hardware placement are evaluated at comparable accuracy.

Critical strengths
  1. Broad coverage of end-to-end agent workflow costs. The paper examines 1,656 configurations spanning seven agentic architectures and nine benchmark/level combinations rather than restricting attention to isolated LLM inference. The resulting accuracy–latency–energy Pareto frontiers give a useful picture of the trade-offs faced by system designers. The component-level analysis of RAG and orchestrated agents is especially valuable because it identifies costs that token counts or single-model benchmarks would miss.

  2. Useful findings about workflow design and hardware placement. The paper provides concrete examples where naive model-call scaling or tool configurations are inefficient, and it shows that the bottleneck can lie in retrieval or orchestration rather than generation. The median-versus-tail comparisons and mixed-GPU experiments also highlight choices that matter in deployment.

  3. High Value for Engineering Practice and Guidance. The paper does not stop at theoretical analysis; it also provides clear guiding principles for real-world AI deployments.These conclusions hold profound practical significance for the industry in building cost-effective, green, and sustainable Agentic AI systems.

Critical weaknesses
  1. Ground-truth-based stopping makes the cascade results hard to apply to deployment. Appendix B states that Multi-Turn AgentChat checks each intermediate response against the benchmark reference answer and stops when the response is correct. A deployed agent usually has no access to that reference answer. This matters because Section 3 attributes part of the efficiency benefit of the cascade to correctly answered queries terminating early, and the main comparison in Figure 2 is framed as evidence that workflow design can reduce cost at matched accuracy. Figure 2 does identify this as a benchmark-only check, but the reported latency and energy may therefore describe an oracle-assisted setup rather than a stopping policy that can actually be deployed. The size of the deployable efficiency gain is uncertain.

  2. The statistical and energy-measurement protocols need more detail. The manuscript reports very large maximum iso-accuracy cost ratios and uses P90/P99 comparisons to motivate deployment decisions, but it is not clear how many independent repetitions support the reported task-level distributions, how variation across runs is handled, or how closely 'iso-accuracy' configurations are matched statistically. On difficult tasks with low accuracy, a change of just a few correctly solved examples could move the estimated frontier.

Questions for authors

Q1: How would the multi-turn and cascaded systems decide to stop without access to the benchmark ground-truth answer? Can you report accuracy, latency, and energy using an implementable stopping criterion, including verification overhead and false-positive/false-negative effects? How does this affect the Figure 2 comparison?

Q2: How exactly is end-to-end energy measured and attributed across GPU inference, CPU retrieval, orchestration, tool execution, and idle time? What are the sampling/integration procedures, measurement boundaries, and repeated-run protocols?

Q3: What is the formal definition of 'iso-accuracy' when constructing and comparing Pareto frontiers? Are configurations matched on identical test examples and exact numbers of correct answers, or within an accuracy tolerance?

1 comment
AuthorsOpenReview ↗

Response to Reviewer bLrw

We thank Reviewer bLrw for their thoughtful and constructive feedback, which has helped us identify opportunities to strengthen our work, and address the key points of the review below.

Multi-Turn AgentChat Termination Criteria:

Applications such as coding already offer practical verification mechanisms, such as unit tests, that can serve as proxies for ground-truth correctness. To demonstrate that our findings also extend to settings without such verifiers, we present additional results in Response to Reviewer (A) using a non-oracle evaluation methodology. Additionally, the compiler-based verification process we use for ground-truth-based stopping in humaneval is analogous to a tool call and typically taking only tens of milliseconds and consuming less than 5J of energy (negligible compared to the cost of agent calls). For other applications, such as MagenticOne, where an orchestrator determines whether a task is complete, we have retained the full execution traces and can use them to evaluate the efficacy of non-oracle verification and stopping strategies. We will incorporate these results into an additional figure in a revised appendix of our paper along with open sourcing our agentic traces.

Hardware Profiling and Energy Evaluation.

In Response to All Reviewers (C), we provide detailed description of our energy evaluation methodology on GPU, CPU energy measurement, including sampling procedures and the number of runs (3) which were used for each configuration. We will gladly include these details in the camera-ready revision.

Iso-Accuracy Definitions.

In our experiments, configurations are defined as iso-accurate if they are within a 1.0% tolerance of the frontier performance with performance measured over the full evaluation set. We provide full details on iso-accuracy evaluation in Response to All Reviewers (D).

Reviewer_GzpvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  • The paper measures how agent design choices affect latency and energy at similar accuracy, across over 1,600 configurations of seven agent systems, from simple multi-turn chat to debate and parallel teams
  • Experiments use small open models such as Qwen3 and Gemma3, from 0.6 to 27 billion parameters, on A6000 and L4 GPUs, with benchmarks such as GAIA, HumanEval, LongBench, SWE-bench Lite and RAG question answering
  • The obtained results suggest that configurations with similar accuracy can differ by up to about 1,000 times in latency and energy, and that cascades of growing models reach the same GAIA L1 accuracy as repeated calls at several times lower cost
  • The paper is essentially a measurement study, and its multi-turn cascades stop as soon as the answer matches the reference, which differs from what a deployed system could/would likely do
Critical strengths
  • The paper compares very different agent designs at matched accuracy, which is more useful for practitioners than the accuracy-only comparisons that dominate the field today, and covers a large space of 356 multi-turn and 432 RAG configurations, plus over 26,000 GPU assignments
  • Its per-agent breakdowns are concrete and point to differences that may be useful in practice. In MagenticOne the orchestrator makes 69% of the calls and takes 85% of the latency, while in SelectorGroupChat the coder makes only 13% of the calls but takes 54% of the latency
  • The RAG study separates retrieval cost from LLM cost. It suggests that searching deeper costs about 12 times more retrieval time for at most half a point of accuracy, and that retrieval takes more than half the total cost in about two thirds of configurations
  • The tail-latency analysis is useful for readers interested in the design of machine learning systems. For repeated calls to a 1.7B model, the 99th percentile of latency reaches 167 seconds against a median of 25 seconds, and which configurations count as best changes with the percentile used
Critical weaknesses
  1. The multi-turn cascades use the reference answer to decide when to stop. Appendix B states that "a correct answer ends the run, while an incorrect answer triggers the next turn". Their accuracy is therefore an oracle pass@k, and their cost drops because solved tasks stop early, which Section 3 credits as one of the "two effects" behind the cascade advantage. These cascades are then compared with MagenticOne, ReAct and parallel teams, which never see the answer

  2. Section 3 changes the thinking mode together with the workflow structure. The repeated baseline is a 1.7B thinking model, while the cascade uses non-thinking models from 0.6B to 8B parameters. The 12.6 times token gap (278 vs 3,497 tokens) is what turning thinking off would produce anyway, so the effect of the structure itself is not isolated

  3. The paper does not seem to explain how energy is measured, although energy is its main contribution. It names neither a measurement tool nor a sampling rate, and does not say which components are counted or how idle GPUs are handled. This makes it difficult to see how claims such as "retrieval, not the LLM, is the bottleneck in 68% of RAG configurations" can hold up, since they compare CPU retrieval energy with GPU inference energy and likely mix two different measurement boundaries

  4. The paper gives no task counts, run counts or confidence intervals. The GAIA L1 accuracies are all multiples of 1/53 or 1/159, which suggests a 53-task set, so an accuracy of 15% carries a standard error of about 5 points, which is high. "Iso-accuracy" is never given a tolerance, and P99 latency over so few tasks essentially reflects the single slowest run

  5. Several headline numbers are difficult to trace back to their source or contradict each other. The abstract's 1,013 times and 1,153 times differences do not appear in any table or figure. The same Section 3 comparison is reported in three ways, as 4.8 times, as 4.9 and 4.6 times, and as 6.3 and 6.1 times, while the stated endpoints (22.8 s vs 3.5 s) give 6.5 times. It is therefore unclear which number comes from where

  6. Section 5 overstates the parallel-agent problem and seems to contradict its own figure. It calls the load imbalance "severe", but the measured idle energy is only 2% and 1.7%, and it is unclear what higher percentages would then mean. The text names Llama-3.1-8B, which does not appear in the Figure 5 legend, and calls Nemotron the slowest path, although in the plot Nemotron is the path that sits idle. The hardware placement gain also comes from searching all assignments with hindsight, which a deployed system could not do

  7. Related work that would make sense to discuss is missing. AI Agents That Matter (2024) already shows that simple retry baselines match complex agents at lower cost, and the Holistic Agent Leaderboard (2025) plots cost and accuracy frontiers across agent setups

  8. Raj et al. is described as studying CPU efficiency "in isolation", but on closer reading it optimizes joint CPU-GPU use. It is also cited twice under two different titles, likely different versions of the same arXiv paper, which cannot be learned from the citations alone and creates further confusion

Questions for authors
  1. How would the Section 3 and Section 6 conclusions change if the reference-answer stopping rule were replaced by something a deployed system could use, such as self-checking, unit tests for HumanEval or a fixed number of turns?

  2. Could the authors add a repeated-call baseline with a non-thinking model, or a cascade of thinking models, to separate the effect of the structure from turning thinking on and off?

  3. What exactly does the energy measurement include, and would the 68% retrieval-bound share hold if GPU and CPU were measured with the same boundary?

  4. Would it be feasible to report bootstrap confidence intervals for the basic quantities, such as accuracy and the P50 and P99 latency, by resampling the GAIA L1 tasks? This would show which of the iso-accuracy comparisons are distinguishable from noise.

  5. How much of the 80% idle-time reduction from mixed GPU placement remains if the assignment has to be chosen before execution, for example tuned on half of the tasks and tested on the other half?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Gzpv

Thank you reviewer for taking the time to thoroughly read our work and the thoughtful comments and the constructive feedback.

Multi-Turn AgentChat Termination Criteria:

We find that our results in Sections 3 and 6 are consistent when evaluating on the HumanEval coding task with verifiable unit tests.

In our experiments, coding benchmarks such as HumanEval already include built-in practical applications for verifiers, making them a natural fit for evaluating our systems. We also add an evaluation of the same taskset without built-in verification and observe performance improvements in those settings as well. For full details, please see Response to All Reviewers (A).

Repeated Calls Across Isolated Non-Thinking and Thinking Structures:

We run additional experiments to show that cascades of non-thinking models in isolation still achieve comparable accuracy to repeated calls to larger thinking models while still providing latency and energy consumption improvements. For full details, please see Response to All Reviewers (B).

Hardware Profiling and Energy Evaluation:

We measure CPU energy using pyRAPL, which captures aggregate CPU and memory energy, and GPU energy using pynvml across multiple trials. For RAG retrieval and inference, these measurements capture the majority of energy consumption on both platforms and provide a reasonable basis for comparison. We provide implementation details in Response to All Reviewers (C) and will include a detailed description of our methodology in a camera-ready appendix.

Parallel Execution:

We agree that the discussion overstates the severity of the energy overhead. We will revise the wording to emphasize that our goal is to address GPU underutilization, where individual agentic paths can experience up to 40% idle time. Our proposed approach leverages older hardware to absorb this otherwise wasted work, reducing idle time, improving overall system utilization, and freeing up newer GPUs for other workloads while extending the useful life of older hardware.

We agree that evaluating the generalizability of mixed GPU placement is important. Offline profiling of representative workloads and characteristics of the inference (i.e. model size and context length) can broadly inform hardware placement decisions. Furthermore, placement decisions do not necessarily need to be fixed entirely before execution to achieve energy and utilization savings. Runtime adaptation mechanisms, such as dynamic voltage and frequency scaling (DVFS), can more dynamically respond to variations in GPU utilization and workload behavior during execution. We will clarify these opportunities and further discuss their implications for practical deployment.

Related Work + Citations:

We thank the reviewer for directing us towards additional references relevant to our work. We will gladly incorporate them in our Related Work for camera-ready revision.

We note that compared to our evaluations, neither Agents that Matter nor the Holistic Agents Leaderboard investigate energy and neither work investigates the tail latency distribution. Additionally, we will correct the citation and highlight that compared to their work our work not only studies the energy efficiency of agentic systems but characterizes the efficiency-accuracy tradeoff frontier.

Consistency and Traceability of Reported Performance Improvements:

Thank you for pointing this out, we agree that the paper does not consistently communicate the magnitude of the reported performance improvements or clearly identify the baseline configurations against which they are measured. We will address this by clarifying the comparisons and explicitly attributing each reported improvement to the corresponding configurations in a camera-ready version.

Evaluation Robustness, Task Counts, and Statistical Confidence

In addition to clarifying our profiling methodology, we will include a detailed table in the appendix documenting the experimental configurations, including task counts, number of trials, model configurations, and other relevant evaluation settings. This will provide greater transparency into our methodology and make the results easier to reproduce.

With regard to statistical confidence of our results, all of our queries tested are evaluated across 3 trials, which we will explicitly clarify in the appendix [A high level version of the results can be seen in the table below]. In the camera-ready version, we will also include an additional figure with a larger sweep of trials, along with statistical confidence intervals for model accuracy across the evaluated queries. This will provide a clearer picture of the variability in our results and the confidence associated with the reported model capabilities.

SystemL1 median stdL2 median std
Single-agent2.381.34
Cascades2.181.34
Naive repeat1.891.34
MagenticOne3.931.78
ReAct2.881.34
Selector1.641.16

Thank you again for your feedback! As the discussion period ends November 18, we would be grateful to know whether our responses address your concerns, and we are happy to clarify any remaining point.

Reviewer_2Cg3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper characterizes 1,656 agentic configurations across model size, repeated calls, retrieval, orchestration, and parallelism. It studies accuracy–latency–energy trade-offs, identifying tool costs, orchestration overhead, tail latency, and synchronization-induced idle energy. The contribution is a systems characterization and deployment guidance rather than a new learning algorithm. It is an interesting topic and requires stronger statistical support, including confidence intervals and repeated trials.

Critical strengths

The study jointly considers accuracy, latency, and energy across multiple structural choices rather than optimizing inference in isolation. Retrieval and orchestrator breakdowns expose overhead that token-only accounting misses. Tail-latency and synchronization analyses identify practical bottlenecks beyond average model-call time. Heterogeneous hardware placement is a concrete opportunity suggested by the measured slack structure.

Critical weaknesses

Multi-Turn AgentChat terminates when the benchmark reference says the answer is correct. Production systems generally lack this signal. Recompute key frontiers using fixed budgets or a realistic verifier, accounting for verifier mistakes and cost. This is an evaluation assumption, not evidence of misconduct. Hardware models alone do not establish whether reported joules include GPU board power, CPU/DRAM, retrieval hosts, idle allocations, and remote tools. Sampling frequency, integration windows, synchronization, and repeated-run variability are necessary to interpret ratios and reproduce results. Repeated calls to a small thinking model are compared with a cascade of larger non-thinking models. This is a relevant configuration comparison but does not isolate structure. Add factorial controls and compare with a single call to the largest cascade model. Equal point estimates on finite benchmarks need not imply equivalent accuracy. Report task counts, paired uncertainty, the tolerance defining iso-accuracy, and uncertainty in Pareto membership. Extremely inefficient configurations do not establish typical production savings. The figure caption, discussion, and opportunity box give differing relative improvements. State whether these reflect distinct populations or summary statistics, and provide a source table.

Questions for authors

all the weak point of above, the paper needs stronger statistical support.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 2Cg3

We thank the reviewer for their thoughtful comments and recognition that our work identifies systems bottlenecks not captured by existing efficiency evaluations.

Multi-Turn AgentChat Termination Criteria:

In our experiments, coding benchmarks such as HumanEval already include built-in practical applications for verifiers, making them a natural fit for evaluating our systems. We also add an evaluation of the same taskset without built-in verification and observe performance improvements in those settings as well. For full details, please see Response to All Reviewers (A).

Hardware Profiling Evaluation Details:

We profile model energy consumption using libraries such as pyRAPL for CPU energy and pynvml for GPU energy across multiple trials. We provide additional implementation details in Response to All Reviewers (C) and will include a more detailed description of our evaluation methodology in the appendix of the camera-ready version.

Repeated Calls Across Isolated Non-Thinking and Thinking Structures:

We provide additional data for isolated non thinking and thinking structure evaluations in Response to All Reviewers (B). In our full set of evaluation results (which we will open source), we have profiled every model size configuration combination that we have tested that allow for a more in depth factorial analysis of these tradeoffs. We will gladly include this in the manuscript for camera-ready revision.

Iso-Accurate Configurations, Relative Performance, and Task Configurations:

We define iso-accurate configurations as those achieving accuracy within 1.0 percentage point of each other (see Response to All Reviewers (D) for further details). Additionally, we will clarify the magnitude of reported performance improvements and explicitly identify the corresponding baseline configurations for each comparison in the camera-ready version. We will also include an additional figure in the appendix analyzing accuracy variability across a larger number of trials to better characterize fluctuations in model accuracy in a camera ready version of the paper.

Production Savings:

We respectfully note that production readiness is not the primary focus of this work; rather, our goal is to characterize the tradeoffs across a range of system configurations. Although some configurations may represent more extreme operating points, we do not believe this makes the resulting tradeoffs irrelevant, as similar conditions can still arise in production settings.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to All Reviewers (A)

Thank you reviewers, for taking the time to carefully read and evaluate our paper and for providing thoughtful and constructive feedback. We appreciate your comments and suggestions, which have helped us identify areas where we can further strengthen our work.

We address the main points of your reviews below:

A) Multi-Turn AgentChat Termination Criteria:

Although ground truth signals are not available for production reasoning tasks such as GAIA, systems with realistic verifiers exist for tasks such as code generation.

In our results in Appendix B on the HumanEval coding task, we demonstrate that our findings with Multiturn Cascades (Section 3) extend to realistic settings with verifiable solutions. Concretely, we find that the 0.6B->1.7B->4B->8B cascade executes with less latency (4.1s vs 19.8s) and utilizes less energy (1.1kJ vs 5.9kJ) than an individual call to the 4B model while maintaining comparable performances of 92.3% and 92.6%, respectively.

Additionally, even without intermediate verification after each llm step, heterogeneous cascades still improve accuracy while reducing latency and energy compared to repeatedly invoking the same large model. For example, the 1.7B->4B->8B->14B cascade achieves comparable accuracy (20.75% vs. 22.64%) while reducing mean latency by 44.5% (44.2s vs. 79.7s) and energy consumption by 45.1% (13.1kJ vs. 23.8kJ) compared to four repeated calls to the 8B model. We will include an updated sweep of these configurations that do not do intermediate checks and their associated accuracy, latency, and energy trade-offs accept the naive checking baseline in the appendix of the revised paper.

CascadeCallModelAcc (%)Lat meanLat P50 / P99 (s)Energy meanEnergy P50 / P99 (J)Tokens P50 / P99
Repeated18Bt13.2123.0223.05 / 143.26,8886,899 / 42,8471,616 / 7,962
2+8Bt18.8747.2043.90 / 347.414,12413,137 / 103,9613,120 / 19,228
3+8Bt20.7561.2961.46 / 477.118,34318,393 / 142,7784,644 / 26,883
4+8Bt22.6479.7086.94 / 625.523,84926,017 / 187,1886,485 / 35,639
No-check14Bt13.2115.5415.13 / 284.84,6104,488 / 84,5291,718 / 24,788
2+8Bt15.0932.4330.39 / 347.09,6659,041 / 103,1653,000 / 28,482
3+14Bt22.6446.9748.48 / 468.814,01414,464 / 139,6064,114 / 32,701
No-check11.7Bt9.435.806.00 / 43.41,6251,682 / 12,1751,448 / 7,460
2+4Bt11.3213.8014.82 / 100.93,9974,225 / 29,3952,605 / 12,800
3+8Bt13.2125.2325.61 / 370.67,4217,493 / 110,1003,873 / 26,874
4+14Bt20.7544.2048.60 / 664.513,09014,428 / 198,0625,152 / 36,466
AuthorsOpenReview ↗

Response to All Reviewers (B)

B) Thinking-model versus cascade comparisons:

We agree that comparing repeated calls to a small thinking model against a cascade of larger non-thinking models does not fully isolate the benefits of the system structure. To address this, we have collected additional comparisons against repeated calls to the largest model and alternative cascaded configurations. For example, the 1.7B->4B->8B->14B non-thinking cascade achieves the same accuracy (22.64%) as four repeated calls to the 8B thinking model while reducing mean latency by 92.0% (6.34s vs. 79.70s) and energy consumption by 92.1% (1.88kJ vs. 23.85kJ). We will incorporate these additional comparisons into the next revision of the paper.

CascadeCallModelAcc (%)Lat meanLat P50 / P99 (s)Energy meanEnergy P50 / P99 (J)Tokens P50 / P99
Repeated18Bt13.2123.0223.05 / 143.26,8886,899 / 42,8471,616 / 7,962
2+8Bt18.8747.2043.90 / 347.414,12413,137 / 103,9613,120 / 19,228
3+8Bt20.7561.2961.46 / 477.118,34318,393 / 142,7784,644 / 26,883
4+8Bt22.6479.7086.94 / 625.523,84926,017 / 187,1876,485 / 35,639
Thinking10.6Bt2.523.062.63 / 16.8545470 / 2,963980 / 3,879
2+1.7Bt9.439.188.93 / 147.62,2192,216 / 40,4442,408 / 24,746
3+8Bt13.2132.6329.18 / 344.59,2628,267 / 99,3583,880 / 35,555
4+14Bt22.6454.2957.64 / 510.815,81616,706 / 149,1595,360 / 41,433
Non-thinking11.7B6.290.110.11 / 1.82626 / 500466 / 958
2+4B11.321.461.28 / 6.8423370 / 1,9681,089 / 2,381
3+8B13.213.423.32 / 16.81,009980 / 4,9671,814 / 4,363
4+14B22.646.347.14 / 48.21,8812,116 / 14,3702,732 / 7,206
Hybrid11.7B6.290.110.11 / 1.92627 / 529471 / 973
2+4B11.321.461.30 / 7.1423377 / 2,0581,098 / 2,335
3+4Bt16.9813.7912.98 / 317.94,0783,834 / 94,3332,808 / 29,869
4+14Bt22.6442.4441.49 / 506.012,65412,353 / 150,6284,375 / 36,634
AuthorsOpenReview ↗

Response to All Reviewers (C)

C) Energy Measurement Methodology:

Below we provide clarifications on the experimental controls and measurement methodology used for energy use evaluations:

To reduce the impact of serving engine startup, we measure the energy use for each configuration after serving 5 warmup instances. Additionally, we address per-request variance by then measuring the per-configuration energy use as the total energy required to process 30 requests following warmup averaged over the total request count with 3 runs per-request.

For GPU inference, energy is measured using the NVML energy counter polled before and after the each llm.generate fn call made by vLLM. For larger models which require multiple GPUs, the energy counters are summed over all devices.

For host CPU energy use, we measure retrieval energy with Intel’s RAPL interface, via the pyRAPL library, which reports cumulative energy for the CPU package and DRAM. Each configuration is measured over 20 consecutive query batches, and we report the mean energy and power per batch.

These hardware-counter-based energy measurement techniques and libraries are well established in recent systems research, including “Where Does the Energy Go? Profiling LLM Agent Inference on Blackwell GPUs” and “From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference”.

The idle power baseline is measured as the change in NVML energy measured over 10 consecutive 60 second period windows, with variance of approximately ±1% on the Ada and ±0.1% on the L4.

For a camera-ready revision, we will gladly expand the appendix to clarify our energy evaluation methodology, including the measurement criteria and open source the code used to collect and process the measurements.

AuthorsOpenReview ↗

Response to All Reviewers (D)

D) Statistical Uncertainty in Accuracy Improvements and Iso-Accuracy:

We acknowledge that aggregate accuracy alone may not fully capture differences in the types of tasks and errors across configurations.

In regards to our methodology for determining iso-accuracy, we identify configurations which attain within 1.0% absolute performance of the performance frontier. Additionally, each example is run over 3 random seeds to reduce per-seed variance.

Given the extensive evaluation data we have retained, we can provide a more detailed analysis of per-task successes and failures to better characterize these differences and clarify the accuracy trade-offs in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.