acceptodds
Under review as a conference paper at ICLR 2027

PACT: Energy-Performance-Aware Commitment Tracking for LLM Serving

Abstract

The power grid must continuously balance electricity supply and demand, making predictable demand important for electricity provisioning. Data center operators therefore often procure electricity under contracts that specify hourly energy commitments, based on which the expected demand can be planned in advance. Deviating from these commitments in either direction can lead to unfavorable settlements or penalties. For data centers serving increasingly dynamic LLM workloads, this creates a new systems challenge: serving decisions directly affect energy consumption, yet the realized consumption must track a precommitted hourly target while preserving request-level SLOs. We formulate this problem as energy-Performance-Aware Commitment Tracking and implement PACT, an LLM serving system built around a two-level controller that periodically revisits the energy vs SLO-performance trade-off and adjusts the serving configuration. A global planner updates near-term energy targets from the remaining hourly commitment and observed consumption, while a local decision maker simulates candidate active-replica and GPU-clock configurations and selects the SLO-feasible action that best matches the target under asymmetric deviation costs. Adaptive action search limits planning overhead while serving proceeds asynchronously. We evaluate PACT against representative latency-first and energy-first baselines, showing its ability to maintain service quality while better aligning realized energy consumption with varying hourly commitments.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM serving, energy-aware scheduling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1eDvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies LLM serving under an hourly energy commitment, where both overuse and underuse are penalized and requests have deadlines. ePACT combines a global planner that spreads the remaining commitment over the remaining minutes with a local controller that uses a calibrated simulator to choose the number of active replicas and the GPU clock. Experiments on 32-GPU H20 and H200 pools compare against vLLM and DynamoLLM. Single-hour runs are on real GPUs, and a simulated 24-hour replay reports about 2.2–2.3% mean hourly deviation versus 11.43% for vLLM at similar SLO attainment.

Critical strengths
  1. The objective is new for LLM serving and clearly defined. Tracking an energy commitment from both sides under deadlines is a different problem from latency-first or energy-minimizing serving, and the settlement and energy accounting are spelled out carefully.
  2. The single-hour experiments run on physical GPUs, and the code is released. On H20 with a fixed commitment, ePACT stays within ±0.1% of the target with SLO attainment within 0.03 pp of vLLM. The controller can clearly track a one-hour target on real hardware.
Critical weaknesses
  1. The baseline numbers are fixed by the protocol. Targets are set as fractions of vLLM's own energy on the same trace (App. C.2), so vLLM's deviation is just a function of the chosen fraction, and every target is reachable by design. For H20 the paper says the targets come from a historical day, but the Table 4 values seem to match evaluation-day vLLM energy.
  2. The 24-hour result doesn't test model error. It is the main claim and comes from CPU simulation, and in the released code the simulated environment and the controller's predictor are the same model. The only prediction error then comes from future arrivals, and the simulator itself is validated only on one replica and one short request slice.
  3. The comparisons don't isolate what ePACT adds. Neither baseline tracks a target, and a simple pacing baseline, such as a power cap set from the remaining commitment, is missing. The ablation suggests most of the gain comes from the pacing rule, and no ablation replaces the simulator search with a simple local rule.
  4. Some details weaken the comparison. DynamoLLM looks under-tuned, with about the same energy as ePACT but 23 pp lower SLO attainment in one H20 setting. Two entries in Table 2 correspond to a 78% target rather than the stated 75%, and all results come from one model and one trace at light load.
Questions for authors
  1. How were the H20 full-day targets computed? The Table 4 values look consistent with evaluation-day vLLM energy rather than a historical day.
  2. In the 24-hour simulation, is the environment the same simulator the controller uses? Results with an independent or perturbed plant model would address my main concern.
  3. Which target was used for each entry in the H20 decreasing-commitment 3.5× column of Table 2? Which entries are the "subsequent corrected entries" mentioned in App. C.2?
  4. The released code has a fast-upscale guard that the paper doesn't describe. Was it on in the reported runs, and how often did it trigger?
  5. How was DynamoLLM tuned?
  6. Could you add a pacing baseline, plus an ablation that keeps the global planner and uses a simple local rule?
2 comments

Weakness 1. The baseline numbers are fixed by the protocol. Targets are set as fractions of vLLM's own energy on the same trace (App. C.2), so vLLM's deviation is just a function of the chosen fraction, and every target is reachable by design. For H20 the paper says the targets come from a historical day, but the Table 4 values seem to match evaluation-day vLLM energy.

Question 1. How were the H20 full-day targets computed? The Table 4 values look consistent with evaluation-day vLLM energy rather than a historical day.

Thank you for checking Table 4. The operator supplies the commitment before control begins. The H20 full-day table used evaluation-day vLLM energy.

Table 4 commitments. The H20 reference was vLLM's energy in each hour of May 11 (the evaluation day) because the public Azure code trace starts May 10 and excludes the previous Saturday, May 4. We chose the lowest fraction tested (75% to 95% in five-point steps) with attainment within 0.5 percentage points of vLLM, defaulting to 100%. We froze the fractions before continuous replay (revised Appendix C.2). The fraction fixes vLLM's deviation. ePACT finished within 5% of target in 22 of 24 H20 hours.

Evidence supporting prior-week reference. We may set commitments in advance based on contracts and expected demand. In two BurstGPT pairs seven days apart, same-hour vLLM energy differed by under 5% (+0.36% at 11:00 on days 52 and 59, −3.30% at 21:00 on days 33 and 40).

Additional prior-week test. Three 32-replica H20 simulations targeted same-hour DynamoLLM energy seven days earlier. The ratio compares test and prior token volumes. Policy cells give attainment / signed energy deviation (%).

Test / prior token ratioCommitment (MJ)vLLMDynamoLLMePACT
0.34311.318152100.000 / +28.471100.000 / −19.617100.000 / −0.021
0.90410.329631100.000 / +40.32087.090 / −8.789100.000 / +0.013
1.77410.370324100.000 / +41.27599.971 / −10.419100.000 / −0.004

Weakness 2. The 24-hour result doesn't test model error. It is the main claim and comes from CPU simulation, and in the released code the simulated environment and the controller's predictor are the same model. The only prediction error comes from future arrivals, and the simulator itself is validated only on one replica and one short request slice.

Question 2. In the 24-hour simulation, is the environment the same simulator the controller uses? Results with an independent or perturbed plant model would address my main concern.

Thank you for proposing a perturbed execution model. The original 24-hour replay shares the predictor's model. In the additional complete-hour test, we froze the predictor and independently increased either execution time or power.

Execution-model error. The 24-hour replay used the same model for prediction and simulated execution. In an additional H20 single-hour simulation, we froze the predictor and independently increased execution-side step time or power (Section 5.4).

Execution conditionAttainment (%)Energy deviation (%)
Nominal98.339−0.053
GPU-step time +10%97.113−0.023
GPU power +10%98.466+4.291

Weakness 3. The comparisons don't isolate what ePACT adds. Neither baseline tracks a target, and a simple pacing baseline, such as a power cap set from the remaining commitment, is missing. The ablation suggests most of the gain comes from the pacing rule, and no ablation replaces the simulator search with a simple local rule.

Question 6. Could you add a pacing baseline, plus an ablation that keeps the global planner and uses a simple local rule?

Thank you for suggesting both comparisons. Against Simple Pacing, ePACT reduced energy deviation by 7.38 percentage points in a hardware replay. With the global planner held fixed in a high-demand replay, ePACT met 1,015 more deadlines than Global + Simple Local.

Additional baselines. On the fixed H20 budget, Simple Pacing overspent and Global + Simple Local missed more deadlines than ePACT. Simple Pacing sets the next minute's power cap from the remaining hourly budget. Global + Simple Local retains ePACT's planner, fixes the clock at 1170 MHz, and selects replica count from fixed power estimates.

Fixed-commitment results. At 3× SLO and 85% of vLLM energy, the entries give deadline attainment / signed energy deviation (%) in 32-replica simulations (Figure 2(a)).

GPUePACTSimple PacingGlobal + Simple Local
H2098.46 / −0.0291.72 / +9.2487.36 / −24.27
H20099.79 / +3.2993.01 / +11.6198.29 / −8.15

Historical pacing targets. At 85% of prior-week vLLM energy in the two BurstGPT hours identified in R3.W1, Simple Pacing overshot by 10.53% and 7.88%. ePACT met every deadline and tracked both targets within 0.051%.

Weakness 4. Some details weaken the comparison. DynamoLLM looks under-tuned, with about the same energy as ePACT but 23 pp lower SLO attainment in one H20 setting. Two entries in Table 2 correspond to a 78% target rather than the stated 75%, and all results come from one model and one trace at light load.

Question 5. How was DynamoLLM tuned?

Thank you for flagging these points.

Why DynamoLLM missed deadlines. DynamoLLM selects the lowest-energy GPU frequency predicted to meet the SLO under the current load. We adopted the five-second interval that the original paper gives as an example for this check (Stojkovic et al., §IV.B). In our full-day experiments, the implemented DynamoLLM estimated load from requests already queued or running at each check. A low frequency could be sufficient for those queued requests, yet too slow for a request that arrived just afterward and was due before the next frequency check.

Baseline and workload coverage. We corrected both Table 2 entries (R3.Q3) and tested a second trace, BurstGPT, including a 137-fold increase in output tokens over the prior week (R3.W1). All runs use Qwen3.8-27B-FP8.

Question 3. Which target was used for each entry in the H20 decreasing-commitment 3.5× column of Table 2? Which entries are the "subsequent corrected entries" mentioned in App. C.2?

Thank you for catching the stale entries. The target being used is 75%, and at the 75% target, DynamoLLM's deviation is +1.56% and Fixed-Config's is +27.32%.

Question 4. The released code has a fast-upscale guard that the paper doesn't describe. Was it on in the reported runs, and how often did it trigger?

Thank you for checking the released implementation. The guard was active for ePACT and could raise capacity between the one-minute planning decisions when deadlines were at risk. It is served as protection for SLO. We have added the description regarding this design to the revised paper (Section 4.3).

Guard activation. The guard was enabled for ePACT and raised capacity on fewer than 0.2% of checks in each 24-hour ePACT replay. It checks deadline pressure every 250 ms and at decision or application boundaries.

GPUChecks that increased capacityTotal guard checksFrequency
H20657347,0970.189%
H200542346,9510.156%
Reviewer_5nqHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies LLM serving within a fixed energy commitment per hour. The method penalizes consuming either too much or too little. The proposed method named ePACT combines a global planner with a local simulator. The global planner distributes the remaining commitment across intervals. The local simulator then supports selecting the number of replicas and GPU clock speeds based on predicted request deadlines. The main contribution of the paper is how to serve requests partly based on how much energy remains to be consumed within the hour. The reported full day results are promising. Mean absolute hourly deviation is 2.16% on H20 and 2.31% on H200. At the same time, SLO attainment remains close to the full capacity vLLM baseline.

Critical strengths
  1. The paper addresses a useful problem and explains it clearly. Tracking an energy commitment can require different serving decisions from minimizing energy consumption. It shows why minimizing energy consumption may not be the best choice when deviations from an hourly energy commitment are penalized.
  2. The controller is easy to understand. It updates the interval target based on the energy already consumed. The method also accounts for draining and standby replicas. This is important because a replica can still consume energy after it stops accepting new requests.
  3. The evaluation keeps unfinished requests and their original deadlines across hours. It also measures energy deviation separately for each hour, so overuse and underuse cannot cancel out in the daily total. The appendix also explains what the energy measurements include and the limitations of the SLO predictions.
Critical weaknesses
  1. The full day results rely on simulation. But the hardware validation covers only a single replica and limited settings. Since the controller and evaluation use the same predictive model, the results may not capture errors between predictions and actual GPU behavior. More hardware validation can help establish whether the reported tracking accuracy holds in practice.
  2. The baseline comparisons need more explanation. DynamoLLM meets the SLO for only about 80% of requests on H20 and 84% on H200 in the full day experiments. Since DynamoLLM is designed to save energy while meeting SLOs, it is not clear why its attainment is this low. Also neither main baseline tries to track the hourly energy commitment.
  3. The evaluation seems to assume that each request’s output length is known when it arrives. The paper should state this assumption clearly, since the output length is usually unknown in a real serving system.
  4. For the adaptive search action, how the selected action compares with the best of all 40 configurations could be included in the results. Basically an offline comparison with exhaustive search would help show how much time it saves and whether it misses better choices.
  5. The comparisons use single runs, and the full day study covers only one trace day. Appendix C also acknowledges that small attainment differences from single runs do not establish statistical significance. Testing additional trace days would help show whether the low energy deviations hold under different workloads.
Questions for authors
  1. How many configurations does the adaptive search check per decision on average? Compared with checking all 40 configurations, how often does it miss a better action? How much does this affect the deviation cost?
  2. The reported results primarily came from either: 1) Real GPU experiment, 2) Simulation to predict request execution and estimate energy consumption. How are the single hour results obtained for the H200 GPUs?
  3. If the average search takes 23.35 seconds within a one minute interval, how does the system handle the delay before applying a decision? What happens if the decision is no longer suitable when the search finishes?
  4. Were exact output lengths known to the controller? If so, then some more experiments on unknown output lengths should be included as well, as known output lengths do not represent realistic scenarios.
  5. How was the DynamoLLM adaptation calibrated and tuned? Why is its SLO attainment much lower than the other methods? Could the authors compare methods at similar SLO attainment and include a simple controller that also tracks the hourly energy target?
  6. Section 4.1 uses one shared GPU clock for all active replicas in a configuration (r, c). Replicas may have different queues and deadline pressure. Allowing different clocks could give finer control over energy consumption and improve tracking while meeting the same SLOs. Could the authors discuss the possible loss from this restriction and the trade-off with the additional planning overhead?
2 comments

Weakness 1. The full day results rely on simulation. But the hardware validation covers only a single replica and limited settings. Since the controller and evaluation use the same predictive model, the results may not capture errors between predictions and actual GPU behavior. More hardware validation can help establish whether the reported tracking accuracy holds in practice.

Thank you for requesting more hardware validation. On real H20, ePACT finished within 5% of its hourly energy commitment in two additional runs below and met at least 98.96% of deadlines in every run.

Additional H20 hardware replays. ePACT finished within 4.4% of target and met at least 98.96% of deadlines in both full-hour runs. Within each May 12 Azure hour, we fixed the requests and commitment and metered full-pool GPU energy.

Azure window (UTC)RequestsePACT attainment (%)ePACT deviation (%)
01:00–02:0012,82298.963+4.371
13:00–14:006,79799.647+3.628

Additional H200 measurement. Across a 90-second run with changing replica counts and clocks, weighted absolute percentage error was 0.923% over two energy intervals and 5.570% over 124 request latencies (Section 5.4).

Weakness 2. The baseline comparisons need more explanation. DynamoLLM meets the SLO for only about 80% of requests on H20 and 84% on H200 in the full day experiments. Since DynamoLLM is designed to save energy while meeting SLOs, it is not clear why its attainment is this low. Also neither main baseline tries to track the hourly energy commitment.

Question 5. How was the DynamoLLM adaptation calibrated and tuned? Why is its SLO attainment much lower than for the other methods? Could the authors compare methods at similar SLO attainment and include a simple controller that also tracks the hourly energy target?

Thank you for raising both questions. DynamoLLM could underestimate demand between GPU frequency checks. We also evaluated Simple Pacing as a baseline that tracks the hourly commitment.

Why DynamoLLM missed deadlines. DynamoLLM selects the lowest-energy GPU frequency predicted to meet the SLO under the current load. We adopted the five-second interval that the original paper gives as an example for this check (Stojkovic et al., §IV.B). In our full-day experiments, the implemented DynamoLLM estimated load from requests already queued or running at each check. A low frequency could be sufficient for those queued requests, yet too slow for a request that arrived just afterward and was due before the next frequency check.

Comparing methods at similar SLO attainment. If we consider recent arrivals that are already due as DynamoLLM makes clock decisions every 5 seconds, it brought attainment within 0.12 percentage points of ePACT's in the H20 simulation. Both methods used the same 16.139 MJ commitment:

MethodDeadline attainment (%)Energy deviation (%)
Arrival-aware DynamoLLM98.22−11.805
ePACT98.34−0.053

Additional pacing baseline. We add a Simple Pacing baseline which sets the next minute's power cap from the remaining hourly budget. At the fixed H20 commitment and 3× SLO, Simple Pacing met 91.72% of deadlines with 9.24% overuse. ePACT met 98.46% with 0.02% underuse (Figure 2(a)). The results can also be shown below, where each entry represents the SLO attainment rate and electricity deviations:

GPUePACTSimple Pacing
H2098.46 / −0.0291.72 / +9.24
H20099.79 / +3.2993.01 / +11.61

Weakness 3. The evaluation seems to assume that each request's output length is known when it arrives. The paper should state this assumption clearly, since the output length is usually unknown in a real serving system.

Question 4. Were exact output lengths known to the controller? If so, then some more experiments on unknown output lengths should be included as well, as known output lengths do not represent realistic scenarios.

Thank you for asking about output lengths. The submitted runs use accurate output length estimations. We have tested the effect of 20% underestimation and overestimation on our performance.

Sensitivity to output-length estimation errors. The original predictor used the accurate output length estimates. In an additional H20 single-hour simulation, we shifted only the estimates by ±20%. The same 26,916 requests retained their actual lengths and deadlines. Both runs finished within 0.03% of the 16.139 MJ commitment (Section 5.4).

Output length supplied to predictorAttainment (%)Energy deviation (%)
Exact98.34−0.053
20% low98.41−0.024
20% high98.41−0.028

Weakness 4. For the adaptive search action, how the selected action compares with the best of all 40 configurations could be included in the results. Basically an offline comparison with exhaustive search would help show how much time it saves and whether it misses better choices.

Question 1. How many configurations does the adaptive search check per decision on average? Compared with checking all 40 configurations, how often does it miss a better action? How much does this affect the deviation cost?

Thank you for suggesting an exhaustive comparison. Adaptive search evaluated 54% fewer configurations and reduced mean wall time by 40%. Both found feasible actions on the same 23 of 30 snapshots.

Search efficiency. Adaptive search checked 18.4 of 40 configurations on average and matched the optimum on 17 of 23 feasible snapshots. Mean planning time fell 40% on the same 30 H20 snapshots with four CPU workers (Section 5.4).

SearchMean configurations checkedMean time (s)95th-percentile time (s)Feasible action foundMinimum cost matched on feasible snapshots
Adaptive18.419.8535.1823/3017/23
Exhaustive4033.0663.0923/3023/23

Cost gap. Across the 23 feasible snapshots, enumeration lowered mean predicted interval cost by 0.0002025 weighted kWh-equivalent. Relative to each interval's energy target, the gap averaged 0.362% and peaked at 3.077%.

Weakness 5. The comparisons use single runs, and the full day study covers only one trace day. Appendix C also acknowledges that small attainment differences from single runs do not establish statistical significance. Testing additional trace days would help show whether the low energy deviations hold under different workloads.

Thank you for asking for additional dates. Three complete BurstGPT hours on different dates test tracking under changing demand, and a physical Azure replay measures run-to-run variation.

Additional trace dates. In one BurstGPT hour on each of three dates, ePACT met all 9,054 deadlines and tracked every target within 0.022%. Each 32-replica H20-profile simulation targeted DynamoLLM's energy from the same hour seven days earlier.

BurstGPT trace day and hourDeadlines met / requestsEnergy deviation (%)
Day 34, 20:003,406 / 3,406−0.004
Day 35, 14:004,184 / 4,184−0.021
Day 40, 15:001,464 / 1,464+0.013

Hardware repeatability. Two H20 runs of the same May 12 Azure 01:00–02:00 UTC workload and commitment differed by 0.20% in metered energy and 0.07 percentage points in attainment.

Question 2. The reported results primarily came from either: 1) Real GPU experiment, 2) Simulation to predict request execution and estimate energy consumption. How are the single hour results obtained for the H200 GPUs?

Thank you for asking. We validated the single-hour simulation results on real H200 GPUs.

Question 3. If the average search takes 23.35 seconds within a one minute interval, how does the system handle the delay before applying a decision? What happens if the decision is no longer suitable when the search finishes?

Thank you for asking about planning delay. Requests continue under the current configuration, and the runtime checks a completed decision against live deadline pressure before applying it.

While search runs. The search runs asynchronously, so requests continue under the current configuration. If deadlines are at risk, the live guard can raise capacity before the search finishes.

When search finishes. The runtime discards an expired decision or one superseded by another control action. It also checks live deadline pressure and can veto a proposed capacity reduction. The runtime applies the decision once it passes these checks (Section 5.4).

Question 6. Section 4.1 uses one shared GPU clock for all active replicas in a configuration (r, c). Replicas may have different queues and deadline pressure. Allowing different clocks could give finer control over energy consumption and improve tracking while meeting the same SLOs. Could the authors discuss the possible loss from this restriction and the trade-off with the additional planning overhead?

Thank you for raising the clock trade-off.

Shared-clock trade-off. Per-replica clocks could improve energy tracking at the same deadline-miss allowance by running replicas with tight deadlines faster and slowing those with more slack. However, with independent clocks, a fixed active-replica count of rr gives 5r5^r clock assignments, making direct search in a large cluster impractical within our 60-second decision interval. We chose a shared clock to keep planning tractable at that cadence. The resulting search space has 40 configurations across eight replica counts and five clocks, and search averaged 23.35 seconds in the H20 run (Section 5.4).

Reviewer_bBCuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to link electricity market pricing schemes to the resource management of LLM serving. Specifically, the paper assumes that LLM serving has a predefined commitment to hourly energy consumption. Both overuse and underuse are subject to penalties. Meanwhile, LLM serving is also constrained by service level objectives (SLOs), and SLO violations incur costs. The paper therefore proposes ePACT to optimise energy consumption and SLO satisfaction for LLM serving. ePACT consists of a global planner that tracks energy consumption and the remaining commitment, and a local decision maker that selects LLM serving configurations to optimise these two objectives.

Critical strengths

The paper uses two GPU pools, each consisting of 32 GPUs, to conduct experiments using a workload trace from a production system. Qwen3.8-27B-FP8 is deployed on each GPU. This provides a reasonable environment for studying the trade-off between energy consumption commitments and SLO violations.

Critical weaknesses
  1. As the authors acknowledge in the Limitations section, the evaluation assumes a hypothetical penalty model for electricity consumption, which does not reflect actual electricity market costs. Meanwhile, SLO satisfaction is not guaranteed.

  2. Although the workload is obtained from a production system, the paper does not describe the inter-arrival times of queries or whether they are scaled. The queries in the request queue appear to be served in a first-come, first-served (FCFS) manner, which does not leverage alternative scheduling strategies to improve the SLO satisfaction rate. Without information about the query inter-arrival times, it is difficult to compare different policies. For example, when the queue is nearly empty, meeting the energy consumption commitment may be difficult for all policies, whereas when the queue is long, query scheduling strategies may matter more than GPU frequency adjustments. Given the current experimental setting, it is difficult to assess how ePACT could be practically deployed.

Questions for authors

see weaknesses

1 comment

Weakness 1. As the authors acknowledge in the Limitations section, the evaluation assumes a hypothetical penalty model for electricity consumption, which does not reflect actual electricity market costs. Meanwhile, SLO satisfaction is not guaranteed.

Thank you for raising both points. It’s up to the deal between the electricity seller and buyer to determine the cost of overuse and underuse, and ePACT prioritizes predicted deadline compliance when the energy target conflicts with SLO.

Electricity contract rates. Operators can set ePACT's overuse and underuse rates according to their needs. In our evaluation, one kWh above the commitment carries twice the penalty of one kWh below it (Appendix C.3). However, different contracts can assign different rates to extra purchases and unused contracted energy. We report signed hourly deviations and energy so that readers can also price a completed run under their own settlement terms.

SLO satisfaction protection. ePACT gives request deadlines priority when they conflict with the energy target. It minimizes deviation cost among configurations that satisfy the predicted deadline-miss allowance. If none qualifies, it selects the configuration with the fewest predicted misses, even when that means exceeding the target. Since a burst after a decision can still cause misses, the system can restore capacity between decisions when deadlines are at risk (Section 4.3). In the 24-hour Azure replay at 3× SLO, ePACT met 99.49% of request deadlines on H20 and 99.94% on H200 (Table 1).

Weakness 2. Although the workload is obtained from a production system, the paper does not describe the inter-arrival times of queries or whether they are scaled. The queries in the request queue appear to be served in a first-come, first-served (FCFS) manner, which does not leverage alternative scheduling strategies to improve the SLO satisfaction rate. Without information about the query inter-arrival times, it is difficult to compare different policies. For example, when the queue is nearly empty, meeting the energy consumption commitment may be difficult for all policies, whereas when the queue is long, query scheduling strategies may matter more than GPU frequency adjustments. Given the current experimental setting, it is difficult to assess how ePACT could be practically deployed.

Thank you for asking about the arrival process. We consider the scheduling strategies orthogonal to our work and all policies see the same arrivals and use the same FCFS scheduler within each experiment. The full-day Azure replay uses the recorded arrival timestamps.

Arrival replay. The full-day Azure experiment preserves recorded inter-arrival times, and all policies use FCFS. The separate single hour uses Azure token lengths in three 20-minute phases averaging 15, 5, and 3 requests/s (Section 5.1).

Queue length and request scheduling. Queue length affects whether the pool can meet deadlines and track the energy commitment at the same time. A nearly empty queue may leave too little work to use the committed energy for serving. When requests accumulate, the scheduler decides which requests run first, and ePACT can add replicas or raise GPU clocks to serve the backlog faster. We hold the FCFS policy fixed across methods to measure how their capacity decisions affect both deadline attainment and commitment tracking.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for your reviews. Our additional experiments compare additional baselines, test prediction and search, and extend the Azure LLM Inference Trace 2024 evaluation to a second public production trace, BurstGPT.

  • Prior-week commitments. In three BurstGPT hours simulated with 32 H20 replicas, ePACT met all 9,054 deadlines and stayed within 0.022% of targets set from same-hour DynamoLLM energy seven days earlier.
  • Serving baselines. In an H20 single-hour simulation, ePACT and arrival-aware DynamoLLM met 98.34% and 98.22% of deadlines, with underuse of 0.053% and 11.805%. In the May 12 Azure H20 hardware replay at 13:00 UTC, ePACT and Simple Pacing both exceeded 99.6% attainment, with overuse of 3.628% and 11.010%.
  • Local selection. With the same global planner, predictive selection raised H20 attainment from 87.36% to 98.46% and cut underuse from 24.27% to 0.02% versus the fixed-clock rule.
  • Search efficiency. Adaptive search checked 18.4 of 40 configurations on average, cutting mean wall time by 40%. Its mean excess predicted cost on 23 feasible snapshots was 0.362% of the interval target.
  • Prediction accuracy. On H200 hardware, weighted absolute percentage error was 0.923% for energy and 5.570% for latency. In the H20 single hour, ±20% output-length errors kept energy within 0.03% of target.

Attainment is the share of requests meeting their original deadlines. Positive and negative deviations denote overuse and underuse, respectively. Revisions made in the paper are in blue.

Best regards,

The Authors of Submission 6024

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.