- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies energy optimization in LLM inference by dynamically tuning GPU frequency through DVFS. The proposed solution requires a latency probe and a given SLO target. It achieves better energy saving through a blackbox controller trained using contextual bandit. Evaluation on an A800 GPU with Azure production LLM inference traces shows that, compared to max frequency setting, HALO reduced total GPU energy by 37.5% in the default setting and by 14.5% to 40.8% across various scenarios.
Critical strengths
- A complete and simple solution to follow
- Evaluations against the max-frequency setting looks promising
Critical weaknesses
- Lack of technical novelty (context bandit solution is well-known and applied to resource management in many scenarios)
- Did not compare with DynamoLLM (which consists of DVFS with a latency SLO target)
- Did not compare with mu-Serve (ATC 2024, which also does DVFS with a latency SLO target)
- Did not separate TTFT from TBT latency metrics; an end-to-end latency probe is too coarse grained
Questions for authors
- How does HALO compare with DynamoLLM?
- How does HALO compare with a threshold-based frequency scaling approach used in muServe (ATC 2024)?
- In most LLM inference cases, SLOs are defined for TTFT and TBT separately. Prefill instances and decode instances are deployed in a disaggregated fashion. How does HALO handle those?
- Most prevalent models are nowadays reasoning models with a lot of reasoning tokens before generating the first output token. How does HALO work with the reasoning models in terms of energy optimization?
- How model paralellism affects energy optimization?