acceptodds
Under review as a conference paper at ICLR 2027

UNDERSTANDING MOE INFERENCE ENERGY CONSUMPTION VIA EXPERT ACTIVATION TRAJECTORY ANALYSIS

Abstract

Large language model inference incurs substantial operational cost because deployed models must serve requests continuously over long periods. Existing work often analyzes request-level inference energy as an aggregate GPU-level measurement. This black-box view is limited for Mixture-of-Experts (MoE) models, where different inputs dynamically activate different experts and form different internal computation paths. This paper studies the inference energy of a Mixtral-style MoE model through a mechanism-oriented lens. We decompose MoE inference into an input-routing-output-energy chain: input prompts induce prefill expert activation trajectories, these trajectories predict later output behavior, output behavior directly drives decode energy, deployment configuration changes the hardware energy coefficient, and correctness determines whether the consumed energy is useful. Across request-level energy traces, reconstructed expert routing features, deployment comparisons, full benchmark correctness results, and expert intervention experiments, we find that output behavior is the strongest direct driver of decode energy; prefill expert activation substantially improves output prediction; expert importance cannot be captured by activation frequency alone; deployment configuration changes energy consumption even when output behavior and correctness are nearly unchanged; and a small number of experts have disproportionate effects on output quality under intervention. These results suggest that MoE inference energy is not merely a single aggregate energy number, but an interpretable and potentially optimizable process shaped by routing, output behavior, deployment, and correctness.

Primary area:
datasets and benchmarks
Keywords:
Mixture-of-Experts, LLM Inference, Inference Energy, Expert Routing, Expert Activation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_n1u1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this work, the authors developed a statistical model to predict MoE inference energy consumption based on output metrics (output length, prefill activations etc.) Using Mixtral-8x7B-Instruct-v0.1 as the main research object, the authors deploy the model on consumer GPUs and Huggingface Transformers algorithms for capturing the energy, quality (against a wide range of benchmarks from different domains, gsm8k, arc_challenge, hellaswag, etc.). Ultimately, the statistical models are developed with the formation process from input-activations-output chain. The authors also proposed the concept of correctness normalization of the energy consumptions.

Critical strengths

I think the authors studied an interesting and important phenomenon, which is the energy consumption of LLMs and its relationship with the architectures, i.e. MoEs. They then propose an interesting concept of input-routing-energy process and develop a statistical model for that. the study is well motivated.

Critical weaknesses

Although the work is well motivated, i personally think the work is significantly lacking in soundness, and the contributions to the community.

Soundness:

  • the main research object with Mixtral-8x7b and Huggingface inference engine, while both of the selections are very outdated and irrelevant to the modern LLM deployment. One could argue that the resources are limited in academic settings, but at least the authors should try small qwen models that can be hosted on consumer GPUs. As for the engine, transformers doesn't really give performant serving setting, and the result from the analysis can be very off considering the sota inference engines like KTransformers, llama.cpp, vllm, sglang etc. no to mention the realistic inference settings like different batch size, etc.
  • writing and presentation: I suspect that the authors heavily use AI tools in visualizing their results/designs, e.g. (a) is missing in Figure 1. and there is no clear explanation of different subfigures in the captions. In Figure 2, there is significant amount of overlapped texts and error bars, making it really hard to interpret the results.

Contributions:

  • although the analysis of MoE routing is novel, i don't think it's the right angle to take for the energy consumption perspective of LLM inference, the deployment settings are highly variant: harware, parallelism, customer traffic, optimizations like prompt caching, quantization, speculative decoding could all play a role in the eventual formula, and the MoE routing part is only one factor of a highly complex system. Fitting a statistical model on the studies features will be quite limited and hard to generalize in realistic settings.
Questions for authors
  1. in line 183: how did you confirm the determinism of the llm inference? did you try to run the inference multiple times for each benchmark, and did you see any variance even if you are doing 0 temperature and 1 top p?
  2. in line 181: why no speculative decoding?
  3. in line 187, huggingface inference is very unrealistic in actual deployment, why not other open source frameworks?
  4. in line 212: "MoE inference on multiple GPUs is not merely a matter of total energy", why do we want to treat the energy consumption per GPU?
  5. have you tried running inference in larger batch size, and longer input prompts?
  6. what's the motivations behind the correctness normalization of the energy consumption?
Reviewer_PgBZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Research problem. The paper argues that LLM inference energy is typically reported as a single, black-box GPU measurement (e.g., total energy or joules per token). Such aggregate numbers hide how energy arises in Mixture-of-Experts (MoE) models, where different inputs activate different experts. The work asks how request-level MoE inference energy is formed and how deployment configuration and output correctness should enter this picture.

Proposed approach. The authors frame MoE inference energy as an input → prefill routing → output behavior → decode energy chain, in which the deployment configuration acts as a hardware energy coefficient and correctness determines whether the consumed energy is useful. No new system or algorithm is proposed; the contribution is an empirical, measurement-based analysis.

Experimental validation. All experiments run Mixtral-8x7B-Instruct (HuggingFace transformers, FP16, greedy decoding) on a server with eight RTX 4090 GPUs. Decode energy is obtained by integrating nvidia-smi power samples (0.1 s interval) over each request's generation window. The evidence comes from three separate suites that do not record every variable for the same requests: (i) a controlled modeling suite with reconstructed prefill routing features, output behavior, and energy; (ii) a full-benchmark suite (GSM8K, MMLU, ARC-Challenge, HellaSwag; roughly 26.6K requests per deployment) executed under three HuggingFace device mappings (automatic, manual contiguous, and round-robin balanced); and (iii) an expert-intervention suite (hard ban, soft downweighting; prefill-only, decode-only, and all-phase) that screens all 256 experts and validates selected ones on a wider benchmark mix that additionally covers MMLU-Pro, BBH, and GPQA.

Main contributions and key findings: ● Output behavior (output length, hit-max, decode time) accounts for most of the decode-energy variance (held-out R² = 0.890). Including input length, task, and deployment-specific terms raises the fit to 0.982 (0.989 with a gradient-boosted tree). ● Prefill expert and routing features lift output-length prediction from R² = 0.351 (input length and task only) to 0.714 with ridge regression, and to as much as 0.830 with a tree model on routing-confidence features. The authors position prefill routing as an early indicator of later energy demand. ● Activation frequency is a poor proxy for expert importance. The paper introduces a correctness-retention utility (paired accuracy loss under intervention), which reaches a Spearman correlation of 0.543 on disjoint validation requests versus −0.037 for activation frequency. Utility is concentrated in a few experts (hard-banning L1_E3 lowers validation accuracy by 19.7 pp, and L31_E2 by 4.6 pp) and varies with task and phase (L1_E3 matters chiefly in prefill). The authors acknowledge that the data do not support a stable ranking of all 256 experts. ● The three 8-GPU deployments reach nearly identical accuracy (≈0.672) and output lengths, yet differ in energy per correct answer (1461–1499 J). Request-matched comparisons reveal nonzero energy gaps with unchanged outputs, which the authors attribute to deployment altering the effective energy coefficient (roughly 673–701 W).

Critical strengths

Strength 1: Large-scale, request-level measurement that links energy to correctness and deployment. Per-GPU energy, output behavior, and correctness are recorded for about 26.6K requests under each of three deployments. Deployments are compared on matched requests: with zero mean differences in output tokens and correctness, mean energy still differs by 11–26 J per request (Table 13). Energy per correct answer is also reported by deployment and discussed by task. This matters because it offers a reusable methodology for evaluating MoE inference energy beyond a single aggregate figure, in line with the paper's datasets-and-benchmarks focus.

Strength 2: Careful expert-intervention design with appropriately narrow claims. Screening (256 experts, 608 requests) is kept separate from validation on disjoint requests (166 overlapping questions removed, leaving 4,372). Bootstrap confidence intervals are reported, and hard/soft as well as prefill/decode/all-phase interventions are compared. The authors concede that the full ranking is unstable and restrict their claims to the two well-supported experts (L1_E3: 18.3 pp, L31_E2: 6.8 pp). As a result, the expert-level conclusions they do draw are credible; the strongest one (L1_E3) also agrees with an independent earlier finding (Su et al., 2025), which, however, makes it a confirmation rather than a discovery (see Weakness 1).

Critical weaknesses

Weakness 1: Limited novelty relative to prior work. Most of the central findings echo known or expected behavior: ● The strongest correctness-critical expert seems to have been identified previously. Su et al. (2025) showed that removing a handful of "Super Experts" severely degrades outputs, and their Super Expert for Mixtral-8x7B-Instruct-v0.1 is Layer 1, Expert 3, which looks identical to L1_E3 here. ● On Mixtral-8x7B, frequency-based expert pruning was shown to underperform other criteria, and even random selection (Lu et al., ACL 2024). ● Wilkins et al. (2024) already modeled energy as a function of input and output tokens, with output tokens dominating (R² = 0.98 for Mixtral-8x7B), and benchmarks such as ML.ENERGY (Chung et al., 2025) routinely measure request-level inference energy. ● Output-length prediction is a well-studied problem in LLM serving (Zheng et al., 2023; S³, Jin et al., 2023; TRAIL, Shahout et al., ICLR 2025, which estimates the remaining length from intermediate-layer embeddings). Since router logits are a linear projection of hidden states, a comparison with such predictors is needed but missing. ● Accuracy-normalized efficiency metrics (e.g., intelligence per watt, Saad-Falcon et al., 2025) and energy comparisons across parallelism strategies (TokenPowerBench, Niu et al., AAAI 2026) have also been proposed.

Apart from ML.ENERGY, the related-work section (Appendix A) omits all of these studies. Without positioning against them, the contribution shrinks to a measurement study of one model on one system.

Weakness 2: The central "routing → output → energy" mechanism lacks direct support in the reported analyses. Appendix D.3 defines a mediation analysis but reports no results. The high R² of the energy models depends partly on decode time (output length alone: R² = 0.721). In Table 6, adding deployment features predicts output length as well as adding routing features, with no explanation. The confidence intervals promised for the deployment comparison are absent. Consequently, the mechanism named in the title, along with two of the five key findings (the predictive value of routing and the deployment effect), remains unestablished.

Questions for authors

Q1. Mediation results (clarification). Could you report Δrouting,total and Δrouting,direct (Appendix D.3) for the controlled modeling suite? These values would show directly whether routing affects energy primarily through output behavior.

Q2. Table 6 (clarification). Why does adding deployment/hardware features to input length and task yield the same R² (0.714) as adding prefill routing features, when outputs are nearly identical across deployments (Tables 10 and 12)? Was a prompt-grouped split used here, and what R² do routing features achieve within each task?

Q3. Phase split (clarification). Is the first generated token assigned to prefill or to decode? Because the answer in multiple-choice tasks is typically the first token, could you report phase-wise intervention results for gsm8k separately?

Q4. Deployment uncertainty (clarification). Could you provide confidence intervals or run-to-run variance for energy per correct answer and for the paired energy differences, as stated in Section 4.4? Which nvidia-smi power field was used? What do the deployment-ID suffixes (e.g., "A2000", "B10000") denote?

Q5. Comparison with prior work (suggestion). Could you compare routing-based length prediction against a hidden-state-based predictor, and discuss Su et al. (2025), who identify Layer 1, Expert 3 of Mixtral-8x7B-Instruct-v0.1 as a "Super Expert"? If this is the same expert as L1_E3, what does the paper contribute beyond that result? Your answer would affect my view of the novelty.

Reviewer_QbUSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The papers propose and empirically evaluate a predictive model for MoE inference energy in a multi-GPU hardware setting, using Mixtral-8x7B-Instruct-v0.1 with a huggingface transformers backend. They decompose energy consumption associated with inference into factors associated with input prompt characteristics, expert activations, and output characteristics. They note that a deployment-aware model of output behavior predicts decode energy (r^2=0.989) while output behavior alone predicts only moderately well (r^2 = 0.893 ± 0.004). The authors also conduct a correctness-constrained energy measurement.

Critical strengths

The problem is well-motivated and a more mature version of this paper is potentially valuable to a broad audience

Critical weaknesses
  • The current experimental setting is too limited to draw broadly useful conclusions. Just a single model, Mixtral-8x7B-Instruct-v0.1, on just the vanilla HF transformers backend, on just a handful of benchmarks -- I understand that it would not be reasonable to ask for a full cross product, but at least some limited validation on additional models or backends, and/or a reasonable effort to describe the concrete implications of remaining limitations (e.g. differences between the mixtral model and other MoEs, differences between transformers and other more realistic backends like vllm, types of use cases that are explicitly not covered by the analysis presented in this paper, etc.)
  • insufficient information presented on existing experiments to draw useful conclusions on the experiments conducted. e.g. r^2 values are reported in the main body, and MSE/MAE-ish metrics are reported somewhere in the appendix in Joules, but without understanding what energy measurements are typical, error is not very useful. Even seemingly high r^2 values like 0.989 can belie a visually obvious skew or substantial outliers at one end or something else that is difficult to see from just the reported numbers. I did not see any useful visuals at all, when imo the most natural thing to do would have been to plot predicted vs actual energy
Questions for authors
  1. re: 4.2, how well does do input sequence characteristics directly predict output energy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.