acceptodds
Withdrawn from ICLR 2027

Learning-Based Dynamic Power Management in LLM Serving with Latency SLOs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The request patterns of real-time LLM services are bursty and vary substantially over time, yet production GPUs are commonly held at their maximum frequency to protect tail-latency targets, a conservative practice that wastes a considerable amount of energy whenever the service has latency headroom. Lowering GPU energy under a tail-latency service-level objective (SLO) is difficult because frequency scaling affects latency in a nonlinear way that depends on queueing and execution dynamics hidden from the operator. We present HALO (Hierarchical Adaptive Latency-Oriented DVFS), which treats the serving system as a black box and reduces GPU energy consumption using only two externally observable signals, the GPU telemetry provided by the standard driver interface and the end-to-end latency of a lightweight probe stream. At the end of every control window, HALO compares the probe p95 latency with the SLO to place the system in one of three coarse operating regimes, and the regime fixes the set of admissible SM-frequency actions. Within this admissible set, we use a LinUCB contextual bandit method to select among small discrete frequency steps using window-level telemetry as its context, so the controller adapts online to non-stationary load without offline profiling, instrumentation of the model, or changes to the serving framework. The bandit learns from an energy-squared-delay cost with a soft penalty on the probe tail, which discourages operation close to the SLO and keeps the learning signal stable despite high latency variance. In a replay of an Azure LLM inference trace against vLLM on an NVIDIA A800, HALO lowers total GPU energy by 37.5% in the default setting and by 14.5% to 40.8% across offered loads, probe SLOs, and three served models, with unchanged throughput and probe p95 below the SLO in 99% of the control windows.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM serving, GPU energy efficiency, DVFS, power management, tail latency, latency SLO, contextual bandit, LinUCB, online learning, black-box control, systems for machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_55tKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies energy optimization in LLM inference by dynamically tuning GPU frequency through DVFS. The proposed solution requires a latency probe and a given SLO target. It achieves better energy saving through a blackbox controller trained using contextual bandit. Evaluation on an A800 GPU with Azure production LLM inference traces shows that, compared to max frequency setting, HALO reduced total GPU energy by 37.5% in the default setting and by 14.5% to 40.8% across various scenarios.

Critical strengths
  • A complete and simple solution to follow
  • Evaluations against the max-frequency setting looks promising
Critical weaknesses
  • Lack of technical novelty (context bandit solution is well-known and applied to resource management in many scenarios)
  • Did not compare with DynamoLLM (which consists of DVFS with a latency SLO target)
  • Did not compare with mu-Serve (ATC 2024, which also does DVFS with a latency SLO target)
  • Did not separate TTFT from TBT latency metrics; an end-to-end latency probe is too coarse grained
Questions for authors
  • How does HALO compare with DynamoLLM?
  • How does HALO compare with a threshold-based frequency scaling approach used in muServe (ATC 2024)?
  • In most LLM inference cases, SLOs are defined for TTFT and TBT separately. Prefill instances and decode instances are deployed in a disaggregated fashion. How does HALO handle those?
  • Most prevalent models are nowadays reasoning models with a lot of reasoning tokens before generating the first output token. How does HALO work with the reasoning models in terms of energy optimization?
  • How model paralellism affects energy optimization?
Reviewer_MGf5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HALO, a GPU frequency controller for LLM serving. It uses probe latency to decide which frequency changes are allowed, then uses a contextual bandit to choose an action from driver telemetry. It runs alongside an unmodified vLLM server. On an A800 with Azure-trace replays, the paper reports 37.5% energy saving in the default setting, with higher workload latency and a small number of probe-SLO violations. It also evaluates different loads, models, and probe targets.

Critical strengths
  • The setup is practical. Only driver telemetry and normal inference requests are needed, so there is no need to change the serving engine or instrument individual kernels. I like this choice for an online controller.

  • The long replay runs gives a useful picture of how the controller behaves as load changes. The authors also report the latency increase, overload cases, and probe work, rather than only showing the energy saving. The energy reduction itself is quite interesting.

Critical weaknesses

The main thing I am missing is a simple adaptive baseline. Comparing only to MaxFreq tells me that there is room to lower the clock, but I cannot tell how much the learning policy adds. For example, how would a controller using the same GREEN/AMBER/RED rules and emergency recovery perform without LinUCB? I do not think many more baselines are needed, but this one seems important.

I also have some concern about the probe. Figure 5 has a window where probe p95 is about 3 s while workload p95 is 116 s. So the correlation alone does not tell us how often the probe misses a bad user-latency window. Table 13 also shows quite a lot of probe output, and the amount differs between policies. The per-token results help, but I am still unsure how much of the gain would remain with a lighter probe stream. These two points are why I am still a little on the rejection side.

Questions for authors
  1. How does HALO compare to a simple controller with the same latency gates but no bandit, at a similar workload-latency budget? This would help show what is gained from learning.

  2. In the existing logs, how often does user latency exceed its target while the probe says GREEN or AMBER? Can you also explain how the different amount of probe work affects the energy comparison? A small check with lower probe traffic would be helpful if feasible.

Reviewer_AQRkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HALO, a learning-based GPU power management approach for LLM serving. It combines latency-based action restrictions with a contextual bandit to dynamically adjust GPU frequency while considering latency SLOs. Experiments using an Azure workload trace on an NVIDIA A800 show energy savings of 14.5–40.8% across different workloads and models.

Critical strengths

The paper addresses an important practical problem in LLM serving. I like the black-box design, which requires no changes to the serving framework or offline profiling. The reported energy savings are substantial, and the evaluation considers different loads, models, and latency constraints. The implementation also appears relatively lightweight.

Critical weaknesses

My main concern is the limited evaluation against existing approaches. The only baseline is a GPU running at maximum frequency, which is not particularly competitive for energy optimization. There are already several learning-based and heuristic DVFS controllers. Without comparing against these methods, it is difficult to judge the benefits of HALO.

The paper also lacks ablation studies. In particular, it is unclear how much of the improvement comes from the contextual bandit rather than the relatively simple latency-based frequency control rules. A rule-based controller using the same three latency zones could potentially achieve similar results.

Another concern is the reliability of the latency SLO. The controller uses a fixed probe workload, but Figure 5 shows a relatively weak correlation between probe latency and actual workload latency (R² = 0.089). The default experiment also increases workload mean latency by 56% and p95 latency by 64%. It is unclear whether the probe-based SLO provides sufficient protection for real user requests.

Finally, the experiments use a single GPU and one workload trace, with each configuration evaluated only once. This limits the evidence for robustness and generality.

Questions for authors

How does HALO compare with existing DVFS controllers, particularly simple rule-based approaches?

Can you provide ablation results showing the contributions of the contextual bandit, latency-based action gating, and reward function?

How reliable is the probe latency as an indicator of actual user latency? Could the controller violate user-facing SLOs even when the probe stays within its SLO?

How sensitive are the results to different workload traces and random seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.