- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies LLM serving under an hourly energy commitment, where both overuse and underuse are penalized and requests have deadlines. ePACT combines a global planner that spreads the remaining commitment over the remaining minutes with a local controller that uses a calibrated simulator to choose the number of active replicas and the GPU clock. Experiments on 32-GPU H20 and H200 pools compare against vLLM and DynamoLLM. Single-hour runs are on real GPUs, and a simulated 24-hour replay reports about 2.2–2.3% mean hourly deviation versus 11.43% for vLLM at similar SLO attainment.
Critical strengths
- The objective is new for LLM serving and clearly defined. Tracking an energy commitment from both sides under deadlines is a different problem from latency-first or energy-minimizing serving, and the settlement and energy accounting are spelled out carefully.
- The single-hour experiments run on physical GPUs, and the code is released. On H20 with a fixed commitment, ePACT stays within ±0.1% of the target with SLO attainment within 0.03 pp of vLLM. The controller can clearly track a one-hour target on real hardware.
Critical weaknesses
- The baseline numbers are fixed by the protocol. Targets are set as fractions of vLLM's own energy on the same trace (App. C.2), so vLLM's deviation is just a function of the chosen fraction, and every target is reachable by design. For H20 the paper says the targets come from a historical day, but the Table 4 values seem to match evaluation-day vLLM energy.
- The 24-hour result doesn't test model error. It is the main claim and comes from CPU simulation, and in the released code the simulated environment and the controller's predictor are the same model. The only prediction error then comes from future arrivals, and the simulator itself is validated only on one replica and one short request slice.
- The comparisons don't isolate what ePACT adds. Neither baseline tracks a target, and a simple pacing baseline, such as a power cap set from the remaining commitment, is missing. The ablation suggests most of the gain comes from the pacing rule, and no ablation replaces the simulator search with a simple local rule.
- Some details weaken the comparison. DynamoLLM looks under-tuned, with about the same energy as ePACT but 23 pp lower SLO attainment in one H20 setting. Two entries in Table 2 correspond to a 78% target rather than the stated 75%, and all results come from one model and one trace at light load.
Questions for authors
- How were the H20 full-day targets computed? The Table 4 values look consistent with evaluation-day vLLM energy rather than a historical day.
- In the 24-hour simulation, is the environment the same simulator the controller uses? Results with an independent or perturbed plant model would address my main concern.
- Which target was used for each entry in the H20 decreasing-commitment 3.5× column of Table 2? Which entries are the "subsequent corrected entries" mentioned in App. C.2?
- The released code has a fast-upscale guard that the paper doesn't describe. Was it on in the reported runs, and how often did it trigger?
- How was DynamoLLM tuned?
- Could you add a pacing baseline, plus an ablation that keeps the global planner and uses a simple local rule?
2 comments
Weakness 4. Some details weaken the comparison. DynamoLLM looks under-tuned, with about the same energy as ePACT but 23 pp lower SLO attainment in one H20 setting. Two entries in Table 2 correspond to a 78% target rather than the stated 75%, and all results come from one model and one trace at light load.
Question 5. How was DynamoLLM tuned?
Thank you for flagging these points.
Why DynamoLLM missed deadlines. DynamoLLM selects the lowest-energy GPU frequency predicted to meet the SLO under the current load. We adopted the five-second interval that the original paper gives as an example for this check (Stojkovic et al., §IV.B). In our full-day experiments, the implemented DynamoLLM estimated load from requests already queued or running at each check. A low frequency could be sufficient for those queued requests, yet too slow for a request that arrived just afterward and was due before the next frequency check.
Baseline and workload coverage. We corrected both Table 2 entries (R3.Q3) and tested a second trace, BurstGPT, including a 137-fold increase in output tokens over the prior week (R3.W1). All runs use Qwen3.8-27B-FP8.
Question 3. Which target was used for each entry in the H20 decreasing-commitment 3.5× column of Table 2? Which entries are the "subsequent corrected entries" mentioned in App. C.2?
Thank you for catching the stale entries. The target being used is 75%, and at the 75% target, DynamoLLM's deviation is +1.56% and Fixed-Config's is +27.32%.
Question 4. The released code has a fast-upscale guard that the paper doesn't describe. Was it on in the reported runs, and how often did it trigger?
Thank you for checking the released implementation. The guard was active for ePACT and could raise capacity between the one-minute planning decisions when deadlines were at risk. It is served as protection for SLO. We have added the description regarding this design to the revised paper (Section 4.3).
Guard activation. The guard was enabled for ePACT and raised capacity on fewer than 0.2% of checks in each 24-hour ePACT replay. It checks deadline pressure every 250 ms and at decision or application boundaries.
| GPU | Checks that increased capacity | Total guard checks | Frequency |
|---|---|---|---|
| H20 | 657 | 347,097 | 0.189% |
| H200 | 542 | 346,951 | 0.156% |
Thank you for checking Table 4. The operator supplies the commitment before control begins. The H20 full-day table used evaluation-day vLLM energy.
Table 4 commitments. The H20 reference was vLLM's energy in each hour of May 11 (the evaluation day) because the public Azure code trace starts May 10 and excludes the previous Saturday, May 4. We chose the lowest fraction tested (75% to 95% in five-point steps) with attainment within 0.5 percentage points of vLLM, defaulting to 100%. We froze the fractions before continuous replay (revised Appendix C.2). The fraction fixes vLLM's deviation. ePACT finished within 5% of target in 22 of 24 H20 hours.
Evidence supporting prior-week reference. We may set commitments in advance based on contracts and expected demand. In two BurstGPT pairs seven days apart, same-hour vLLM energy differed by under 5% (+0.36% at 11:00 on days 52 and 59, −3.30% at 21:00 on days 33 and 40).
Additional prior-week test. Three 32-replica H20 simulations targeted same-hour DynamoLLM energy seven days earlier. The ratio compares test and prior token volumes. Policy cells give attainment / signed energy deviation (%).
Thank you for proposing a perturbed execution model. The original 24-hour replay shares the predictor's model. In the additional complete-hour test, we froze the predictor and independently increased either execution time or power.
Execution-model error. The 24-hour replay used the same model for prediction and simulated execution. In an additional H20 single-hour simulation, we froze the predictor and independently increased execution-side step time or power (Section 5.4).
Thank you for suggesting both comparisons. Against Simple Pacing, ePACT reduced energy deviation by 7.38 percentage points in a hardware replay. With the global planner held fixed in a high-demand replay, ePACT met 1,015 more deadlines than Global + Simple Local.
Additional baselines. On the fixed H20 budget, Simple Pacing overspent and Global + Simple Local missed more deadlines than ePACT. Simple Pacing sets the next minute's power cap from the remaining hourly budget. Global + Simple Local retains ePACT's planner, fixes the clock at 1170 MHz, and selects replica count from fixed power estimates.
Fixed-commitment results. At 3× SLO and 85% of vLLM energy, the entries give deadline attainment / signed energy deviation (%) in 32-replica simulations (Figure 2(a)).
Historical pacing targets. At 85% of prior-week vLLM energy in the two BurstGPT hours identified in R3.W1, Simple Pacing overshot by 10.53% and 7.88%. ePACT met every deadline and tracked both targets within 0.051%.