acceptodds
Under review as a conference paper at ICLR 2027

ECO: Energy-Oriented Configuration Optimization for Attention–FFN Disaggregated LLM Serving

Abstract

Efficient large language model serving requires reducing energy while preserving latency and throughput. Attention–feed-forward network disaggregation exposes separate resource and operating choices, but their interactions create a large configuration space. Existing profile-based approaches require substantial offline measurements and need additional calibration data when deployed on new hardware. Untested configurations may also fail execution or violate service targets. We present Energy-Oriented Configuration Optimization (ECO), a framework for jointly tuning deployment and GPU operating settings within a limited measurement budget. ECO combines a calibrated stage-time and energy model with Gaussian-process corrections and cost-aware constrained Bayesian optimization. It separately models execution validity and service feasibility, selecting measurements according to expected energy improvement and GPU-time cost. The lowest-energy measured feasible configuration is then frozen for serving. Experiments with Qwen and DeepSeek cover A100 and A6000 GPUs. Across eight A6000 and four four-GPU A100 scenarios, ECO reduces serving energy by 44.4% and increases output token rate by 23.8% on average relative to Default while meeting service targets. Across eight A6000 scenarios, its selected feasible energy averages 33.1% below Bayesian optimization and 25.8% below genetic search.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large language model serving, mixture-of-experts, attention–FFN disaggregation, energy efficiency, constrained Bayesian optimization, GPU configuration optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Rq2GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to find configurations for attention–FFN disaggregated LLM serving that satisfy latency and throughput SLOs while reducing total GPU energy consumption under a limited budget of physical measurements. The search space includes GPU counts, parallelism degrees, DBO settings, and the frequencies and power caps of the two GPU groups.

ECO constructs an energy prior from measurement-calibrated stage-time and pipeline models, learns prediction residuals with a Gaussian process, and selects the next trial using constrained Bayesian optimization that accounts for SLO feasibility, execution success, and measurement cost. It returns the lowest-energy feasible configuration among those actually measured. Its main contribution is an energy modeling and search method tailored to the configuration hierarchy and pipeline dependencies of AFD.

The evaluation covers 16 scenarios across two models and two GPU platforms, including end-to-end evaluation on independent request sets, search-efficiency comparisons, and ablations of the energy model, joint tuning, and cost-aware strategy.

Critical strengths
  1. The paper addresses a practically relevant AFD energy-optimization problem through joint search over deployment structures and operating controls. Its formulation accounts for service constraints, execution failures, and measurement costs.

  2. Combining a stage-and-pipeline energy prior with GP residual learning helps guide search under a limited measurement budget. The evaluation includes two models, two hardware platforms, and independent request sets, while the ablations provide support for the roles of the main components.

Critical weaknesses
  1. A comparison with the closest AFD optimization approach is missing. Section 6 identifies AFlex as the closest work, but the BO/GA comparisons and component ablations do not establish how much additional energy saving ECO achieves over a specialized AFD optimization method under the same measurement budget.

  2. The load pressure underlying the energy results is unclear. Section 5 notes that the selected eight-GPU Qwen configurations have less than 1% headroom above the throughput SLO floor, but this does not establish their proximity to maximum sustainable throughput. The experiments cover 4–16 requests/s without locating these loads relative to that capacity, leaving it unclear whether the reported energy savings persist near the serving capacity limit.

Questions for authors
  1. Regarding Weakness 1: With profiling, calibration, and search costs accounted for consistently, could you compare the energy of the final feasible configurations against AFlex or a comparable implementation under the same budget?

  2. Regarding Weakness 2: Where do the evaluated loads lie relative to the maximum sustainable throughput under the stated latency SLOs? How do the energy savings change as load approaches that boundary?

Reviewer_aWcWOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

ECO jointly searches GPU allocation, parallelism, DBO thresholds, frequencies, and power caps for attention–FFN disaggregated serving under a limited physical-evaluation budget. It combines a stage-and-pipeline energy prior with GP residual correction and selects trials using energy improvement, feasibility, execution success, and evaluation cost. The selected configuration is frozen before evaluation on disjoint requests. The systems problem is meaningful and the ablations are encouraging, but the incremental contribution, predictive calibration, and net deployment benefits need stronger validation.

Critical strengths
  1. The paper identifies an important coupling between deployment structure and operating controls in AFD.
  2. The evaluation protocol takes useful steps to control selection bias and account for search cost.
Critical weaknesses

The current comparisons do not sufficiently establish the additional value of the AFD-specific mechanisms over the closest prior work. Feasibility-weighted EI and cost-aware acquisition are established ideas [1,2]. AFlex jointly handles A/F provisioning, frequency control, and pipelining [3], while SCOOT addresses known and hidden constraints in serving-engine tuning [4]. The paper acknowledges these connections, but the principal quantitative comparisons remain Default, BO, and GA. Please add matched profile-guided or serving-tuning baselines, controlling the backend, adjustable parameters, and probing budget. Ablations should separately isolate the pipeline prior, structural probes, and candidate screening, with sensitivity analysis for reasonable BO configurations.

References:

[1] Snoek, J., Larochelle, H., & Adams, R. P. Practical Bayesian Optimization of Machine Learning Algorithms. NeurIPS 2012. arXiv DOI:10.48550/arXiv.1206.2944。arxiv.org

[2] Gelbart, M. A., Snoek, J., & Adams, R. P. Bayesian Optimization with Unknown Constraints. UAI 2014.

[3] Basit, O., Liu, Y., Kong, Z. J., & Hu, Y. C. DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS. 2026.

[4] Cheng, K., et al. SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines. The ACM Web Conference 2025.

Questions for authors

How consistently does ECO outperform an optimizer with identical constraint/cost models and probing budgets but without pipeline mechanism information?

Reviewer_K8krOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ECO, an energy-aware configuration selection framework for Attention–FFN Disaggregated (AFD) LLM serving that searches hierarchical allocation, parallelism, and hardware operating settings under a strict 16-measurement budget, which pairs a structure-aware stage-and-pipeline energy prior with residual Gaussian process modeling and cost-aware Bayesian optimization to achieve substantial energy reductions across multi-GPU setups.

Critical strengths
  1. The framework injects domain-specific pipeline execution structures into the surrogate prior rather than relying on naive black-box exploration, which allows the Gaussian process to fit only residual errors and makes optimization practically viable within a tight 16-measurement budget.

  2. The empirical evaluation adheres to a rigorous offline protocol where SLO targets are predetermined and evaluated on strictly disjoint held-out request streams, which provides reproducible validation across distinct GPU architectures and disaggregated workloads.

Critical weaknesses
  1. The empirical claims suffer from an apparent contradiction in execution methodology and an absence of variance reporting. While Section 5 mentions repeating searches across multiple random seeds, Appendix B.5 explicitly clarifies that all reported comparisons rely on a single deterministic seed (seed 0) per arm with hardcoded RNG seeds. The headline improvements (e.g., 40.5% energy reduction and 20.7% throughput gains) are consequently single-run point estimates devoid of error bars or confidence intervals. Compounding this issue, the paper never reports the numerical fidelity of the stage-and-pipeline prior against physical ground truth, leaving it unverified whether the optimization gains stem from the engineered prior or purely from early heuristic pruning.

  2. The static operational boundaries of the selected configurations lack headroom verification and drift resilience. The evaluation concedes that held-out replays omit load shifts, even though Finding 3 reveals that the selected eight-GPU configurations operate with less than 1% throughput margin. Under real-world bursty arrival rates, such razor-thin headroom creates immediate risks of cascading SLO violations that a 0/1 static feasibility check fails to capture. Furthermore, the paper provides no analytical formulation regarding when an offline static profile must be invalidated and replanned, omitting the computational overhead required to recalibrate the prior when traffic distributions shift.

  3. The baseline comparisons are structurally skewed and omit critical overhead accounting. The competitive BO baseline has its hyperparameter fitting explicitly disabled and kernel length scales frozen (Table 4), while comparison against untuned defaults inflates relative efficiency gains without testing simple frequency sweeps at fixed allocations. The evaluation also overlooks dynamic provisioning baselines (e.g., Flying Serving, AFD-Ledger) and fails to report the aggregated GPU-hours expended across the 16 offline profiling steps, obscuring the break-even serving duration needed to amortize offline search costs.

Questions for authors
  1. Questions For AuthorsCould you clarify the discrepancy between the multi-seed claim in Section 5 and the single-seed specification in Appendix B.5, and provide error bars or seed variance for the primary energy reduction metrics alongside quantitative prior-versus-measured error metrics?

  2. How do the selected configurations behave under dynamic load shifts (e.g., ±10–20%\pm 10\text{--}20\% arrival rate variations), and what are the empirical SLO violation rates on held-out traces given the <1%<1\% throughput headroom in eight-GPU setups?

  3. Across the evaluated workloads, during what fraction of serving time did the chosen power caps actually bind, and would replacing coarse power capping with SM clock locking alter the effective candidate search space or achieved efficiency?

  4. What is the total search overhead in GPU-hours across the 16 physical measurement steps, and what is the estimated break-even serving volume required to amortize this profiling cost relative to a properly tuned BO baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.