- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
In this work, the authors developed a statistical model to predict MoE inference energy consumption based on output metrics (output length, prefill activations etc.) Using Mixtral-8x7B-Instruct-v0.1 as the main research object, the authors deploy the model on consumer GPUs and Huggingface Transformers algorithms for capturing the energy, quality (against a wide range of benchmarks from different domains, gsm8k, arc_challenge, hellaswag, etc.). Ultimately, the statistical models are developed with the formation process from input-activations-output chain. The authors also proposed the concept of correctness normalization of the energy consumptions.
Critical strengths
I think the authors studied an interesting and important phenomenon, which is the energy consumption of LLMs and its relationship with the architectures, i.e. MoEs. They then propose an interesting concept of input-routing-energy process and develop a statistical model for that. the study is well motivated.
Critical weaknesses
Although the work is well motivated, i personally think the work is significantly lacking in soundness, and the contributions to the community.
Soundness:
- the main research object with Mixtral-8x7b and Huggingface inference engine, while both of the selections are very outdated and irrelevant to the modern LLM deployment. One could argue that the resources are limited in academic settings, but at least the authors should try small qwen models that can be hosted on consumer GPUs. As for the engine, transformers doesn't really give performant serving setting, and the result from the analysis can be very off considering the sota inference engines like KTransformers, llama.cpp, vllm, sglang etc. no to mention the realistic inference settings like different batch size, etc.
- writing and presentation: I suspect that the authors heavily use AI tools in visualizing their results/designs, e.g. (a) is missing in Figure 1. and there is no clear explanation of different subfigures in the captions. In Figure 2, there is significant amount of overlapped texts and error bars, making it really hard to interpret the results.
Contributions:
- although the analysis of MoE routing is novel, i don't think it's the right angle to take for the energy consumption perspective of LLM inference, the deployment settings are highly variant: harware, parallelism, customer traffic, optimizations like prompt caching, quantization, speculative decoding could all play a role in the eventual formula, and the MoE routing part is only one factor of a highly complex system. Fitting a statistical model on the studies features will be quite limited and hard to generalize in realistic settings.
Questions for authors
- in line 183: how did you confirm the determinism of the llm inference? did you try to run the inference multiple times for each benchmark, and did you see any variance even if you are doing 0 temperature and 1 top p?
- in line 181: why no speculative decoding?
- in line 187, huggingface inference is very unrealistic in actual deployment, why not other open source frameworks?
- in line 212: "MoE inference on multiple GPUs is not merely a matter of total energy", why do we want to treat the energy consumption per GPU?
- have you tried running inference in larger batch size, and longer input prompts?
- what's the motivations behind the correctness normalization of the energy consumption?