acceptodds
Under review as a conference paper at ICLR 2027

E1: A First-Order Energy Metric for AI Model Architectures

Abstract

Energy is a primary constraint on AI inference at scale, yet models are designed and compared on accuracy alone. A trustworthy energy number requires an optimized implementation, which arrives long after the architecture is fixed, and only for the ideas that well-resourced teams choose to engineer. We introduce E1, an infrastructure-agnostic energy metric computed from unoptimized PyTorch code. E1 lowers the model to a graph of einsums, searches the tiling and fusion schedules of that graph on an abstract accelerator architecture with a finite on-chip buffer, and sums the energy of the arithmetic operations and off-chip data movement. The resulting E1 score represents what an ideally optimized implementation of the model could achieve, without any system optimization work by the developer. We validate E1 against energy measured on NVIDIA H100 GPUs for individual kernels, production LLM-serving kernels across implementations and across two years of their development, and for full model forward passes. E1 tracks measured energy over three orders of magnitude; optimized kernels land - above it and eager PyTorch about an order of magnitude higher; and when one mixture-of-experts model runs under two implementations whose measured energy differs by , E1 does not move. Our target is a reliable, infrastructure-agnostic metric that will enable fair accuracy-energy studies. We plan to open-source E1.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Energy, Systems, Infrastructure, Energy Estimation, Energy Efficiency, Analytical Modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cWv7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces E1, a metric that estimates the energy an optimized implementation of a model could achieve, computed directly from unoptimized PyTorch code with no GPU and no kernel tuning. The pipeline lowers a model to an operator graph, then an einsum graph, then searches the tiling and fusion mapspace of an abstract accelerator for the schedule that minimizes total energy against a small calibrated energy table. The paper validates E1 against energy measured on an NVIDIA H100, the same GPU the table was calibrated on, across microbenchmarks spanning three orders of magnitude, KernelBench tasks, production kernels for LLM serving across competing implementations and across five real historical versions of the same kernel, and full forward passes of a mixture-of-experts model, two dense models, and a recurrent block.

Critical strengths
  1. Section 3.4's mixture-of-experts test confirms on real hardware that an "8x" board energy gap between two implementations is entirely idle power, with energies agreeing within 3% once idle power is subtracted while E1 stays unchanged. Table 2's five real historical versions of FlashInfer's kernel give the same kind of externally verifiable result, with measured energy moving "-6% to +10%" across two years of development while E1's score does not move.
Critical weaknesses
  1. Section 2.2 motivates E1 as "infrastructure-agnostic" across GPUs, TPUs, and RDUs, but every validation experiment measures against the same H100 used to calibrate the energy table. The hard part of that claim, generalizing across hardware families instead of implementations on one GPU, is asserted but not tested.

  2. Section 2.4 excludes launch overhead entirely because "launch overhead disappears with fusion or CUDA Graphs," supported by one worked example (Section 3.4) where the underlying computation can be merged into a single grouped kernel. This does not establish the exclusion for architectures where each expert's or branch's computation cannot be grouped at all, which is exactly the kind of architecture comparison E1 is meant to support.

Questions for authors
  1. Has E1 been checked against measured energy on any accelerator other than the H100 the energy table was calibrated on, and would the same tracking hold if a different GPU generation or a non-GPU target were calibrated instead?

  2. Section 2.4 excludes launch overhead because it disappears with fusion or CUDA Graphs. Does that hold for architectures where each unit's computation cannot be merged into one grouped kernel, such as routing to experts of heterogeneous shape, where many small launches may be an architecturally forced cost instead of an artifact of immature kernel engineering?

  3. The abstract states optimized kernels land "2-4x" above E1, narrower than the 2.3-6x range reported elsewhere in the paper (Appendix C, Figure 5). Could the abstract's range be reconciled with the fuller spread the paper itself reports?

  4. Section 2.2 names SRAM-only designs as explicitly out of scope because they lack an off-chip memory boundary. Does the same limitation apply to neuromorphic or other non-von-Neumann accelerators, and should the claim of infrastructure agnosticism name that exclusion too?

1 comment
AuthorsOpenReview ↗

Response to Reviewer cWv7

Thank you for the review, and for highlighting that E1's score does not move while the implementations of the same model or kernel change.

W1/Q1. "Infrastructure-agnostic" and other accelerators. We have added a clarification in Section 2.2 of what makes E1 infrastructure-agnostic, as its hardware target does not model a microarchitecture like other tools do. However, to meaningfully compare a score with real measurements we need reference numbers. We took those from the H100, an accelerator available to us and widely deployed. Calibrating another GPU generation or a non-GPU target would replace the reference numbers and leave the rest of E1 unchanged. We have added Appendix H to show how E1 scores shift as the ratio between compute and memory cost changes. Still, calibration is optional: one can also set these numbers directly, for example to weight memory accesses more heavily.

W2/Q2. Launch overhead when experts cannot be merged. E1 focuses on the efficiency of the model. Over time, systems optimizations have removed overheads that are not tied to it: CUDA graphs replay a launch sequence without the host, grouped GEMM kernels (e.g., in CUTLASS) run matrix products of different shapes in one launch, and megakernels such as MegaMoE run a whole mixture-of-experts layer as a single kernel. We have added an experiment on Qwen3-30B-A3B without merging any experts (Section 3.4, Figure 7c; Appendix J): with one kernel sequence per expert, the routes of the prompt fixed, and the launches replayed as a CUDA graph, board energy drops from 123 J to 20.1 J, against 14.1 J for the grouped kernel.

Q3. 2-4x in the abstract. Thank you for noticing this inconsistency. We have corrected the abstract to 1.9-6x (the added full-model experiments reach 1.9x; see Section 3.4, Appendix J).

Q4. Non-von-Neumann designs. Thank you for the suggestion. We have added analog in-memory, neuromorphic, optical and other non-von-Neumann designs to Section 2.2 as outside the scope, next to the SRAM-only designs.

Reviewer_VVH9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces E1, an infrastructure-agnostic, first-order analytical energy metric designed to estimate the dynamic energy efficiency potential of an AI model architecture directly from unoptimized PyTorch code—without requiring a GPU or manual kernel engineering. E1 operates in four stages: (1) tracing a PyTorch forward pass into an operator graph via Solar, (2) lowering operators into an einsum graph, (3) searching over tiling and cross-operator fusion schedules using AccelForge on a minimal abstract accelerator ("Snowcat", modeled with unbounded off-chip main memory, a finite 50 MB on-chip global buffer, and a compute unit), and (4) multiplying arithmetic operation counts (MACs per bit-width) and off-chip memory traffic (bytes read/written across the HBM boundary) by a calibrated hardware energy table.

The authors validate E1 against empirical board and idle-subtracted energy measurements on an NVIDIA H100 across four levels: (1) operator microbenchmarks (§3.1), (2) KernelBench L1 single operators and L2 short fused programs (§3.2), (3) production LLM-serving kernels from FlashInfer across human/agent implementations and across six historical commits over two years (§3.3), and (4) full-model forward passes on dense LLMs (Qwen3-8B, Gemma 4 12B), a sparse MoE model (Qwen3-30B-A3B), and a recurrent Mamba decode block (§3.4, Appendix G). Results show that E1 tracks optimized empirical energy across three orders of magnitude, remains invariant when two expert implementations of the same MoE model differ by 8× in measured board energy, and reveals when further kernel-level memory optimization has hit diminishing returns.

Critical strengths
  1. Decoupling Architectural Energy Potential from Software Implementation Maturity: A long-standing bottleneck in efficient AI research is that fair energy evaluation requires months of hardware-specific kernel engineering, causing novel architectures running in eager PyTorch to be unfairly penalized while FLOP counts completely miss memory movement costs (which can differ by 1700× per op between element-wise additions and large GEMMs). By pairing an einsum graph representation with an exact tiling-and-fusion mapspace search over a simple two-level memory boundary, E1 provides a practical, CPU-computable lower bound on dynamic energy. The Qwen3-30B-A3B experiment (§3.4) compellingly demonstrates this invariance: while a naive per-expert loop consumes 8× more measured board energy than a grouped-GEMM implementation, E1 assigns the exact same score (3.45 J) to both.

  2. Comprehensive Multi-Scale Validation and Actionable Engineering Insights on Optimization Headroom: The experimental analysis is thorough and well-structured across scales—spanning microbenchmarks, KernelBench, production FlashInfer kernels, and full pretrained models. Particularly insightful is the longitudinal analysis of FlashInfer’s fused add+RMSNorm kernel over two years (§3.3, Figure 5b) and the traffic-bound analysis in Appendix F: E1 does not merely rank implementations, but diagnoses why a gap exists and when to stop optimizing. Showing that the initial operator fusion captured a 4.8× energy reduction while five subsequent commits over two years shifted energy by only -6% to +10% (because memory traffic had already reached the E1 minimum, leaving only launch and idle overheads) provides clear, practical guidance for both kernel engineers and automated kernel-generation agents.

Critical weaknesses
  1. Lack of End-to-End Validation across Context Lengths and Batch Regimes: All full-model evaluations in §3.4 are fixed to a single, very short input regime: batch size = 1 and a 128-token prefill (and 1-token decode for Mamba in Appendix G). In this small-batch, short-context regime, energy is overwhelmingly dominated by reading static model weights from HBM (73% in dense FFN blocks and 89% in MoE expert weights), while the attention core, RoPE, and Q/K norms together account for only 0.4%–1% of total energy. Because of this, the authors do not even run the mapper on the attention core in full models, estimating it from raw byte counts instead (Appendix D). However, in realistic long-context prefill (e.g., 8K–128K tokens) or large-batch decoding with large KV caches, the energy bottleneck shifts dramatically from weight reads to attention computation and KV-cache memory traffic—precisely where architectural innovations (e.g., GQA, MLA, linear/hybrid attention) compete. Evaluating full models across varying sequence lengths is essential to prove E1's utility for modern architectural comparisons.

  2. Disconnect Between Single-Kernel Optimization (§3.3) and Full-Model Energy Propagation (§3.4): While §3.3 demonstrates up to 22× energy savings on individual fused kernels (e.g., FlashInfer add+RMSNorm) and Appendix G shows gains from torch.compile on a single Mamba block, §3.4 evaluates full models exclusively under the default HuggingFace Transformers implementation. Aside from swapping the MoE expert loop (which already accounts for 89% of the model's E1 score, and where the 8× measured difference comes almost entirely from idle power rather than dynamic memory savings), the paper never tests how optimizing individual sub-components or integrating fused kernels (e.g., via FlashInfer, torch.compile, or vLLM/TensorRT-LLM) propagates to end-to-end model energy reduction and converges toward the full-model E1 bound. Furthermore, because full models are segmented at materialized checkpoints and summed linearly, composite multi-kernel execution effects (e.g., cumulative launch overheads across hundreds of small operators or stream contention) remain outside the model.

Questions for authors
  1. Clarification / Additional Experiments on Context Length Scaling: How does the E1 breakdown and the gap to measured H100 energy evolve as sequence length scales from 128 tokens up to 4K, 32K, or 128K tokens on Qwen3-8B, Gemma 4 12B (especially given its alternating local/global attention), and Qwen3-30B-A3B? At what sequence length does the attention core overtake weight reads in E1, and does segmentation / skipping AccelForge mapping on the attention core (Appendix D) still hold without introducing large errors in long-context regimes?

  2. Clarification on End-to-End Propagation of Kernel Optimizations: Can you provide an experiment on one of the full models in §3.4 comparing the measured energy of (a) eager HuggingFace Transformers, (b) torch.compile / FlashInfer-enabled execution, and (c) a production serving engine (e.g., vLLM or TensorRT-LLM) against the E1 score? Showing that progressively fusing kernels in a full model systematically closes the gap toward the E1 lower bound would bridge the current disconnect between §3.3 and §3.4.

  3. Suggestion for Future Work — Algebraic Rewrite Layer and Kernel Design Guidance: Since FlashAttention-style online softmax requires an algebraic graph rewrite before tiling/fusion search, how sensitive is E1's ranking of attention variants if such rewrites are absent? Additionally, since AccelForge already finds an optimal tiling and fusion schedule to compute E1, could E1 export this schedule as actionable hints (e.g., recommended fusion boundaries and tile sizes) to guide human engineers or LLM kernel-writing agents?

1 comment
AuthorsOpenReview ↗

Response to Reviewer VVH9

Thank you for the concrete suggestions, and for highlighting how E1 separates an architecture's energy potential from the maturity of its implementation. We ran the three models on the H100 at prompt lengths from 128 to 32K tokens (131K where the model supports it) and under implementations from eager PyTorch to vLLM, and compared every run with its E1 score.

W1/Q1. Context length and batch regimes. We have added the suggested experiment in Section 3.4 (Figure 7a, 7b) and Appendix J (Tables 3 and 4). The attention core is now mapped at every length; for Qwen3-8B it overtakes the weight reads near 2.7K tokens. The gap to measured energy stays stable as the prompt grows: served by vLLM, all three models are at 1.9-2.6x E1 at every length measured, from 128 to 131K tokens (Table 3). Segmentation still holds at long prompts: all activation traffic at segment boundaries is 6-10% of a score at 32K tokens, which bounds what it can cost (Appendix J).

W2/Q2. From kernel optimizations to the full model. We have added the suggested experiment in Section 3.4 (Figure 7c) and Appendix J (Table 5). The gap closes toward E1: a 128-token Qwen3-8B prefill goes from 3.5x E1 in eager PyTorch to 2.3x with torch.compile, 1.9x with CUDA graphs added, and 2.0x under vLLM.

Q3. Attention without the algebraic rewrite. How sensitive E1 is to an algebraic rewrite depends on whether the rewrite unlocks a schedule with less memory traffic or less arithmetic than E1 finds without it. The missing rewrite is a limitation of the mapper, not of the metric: a mapper that also searched over such rewrites would extend E1 without changing its energy model.

Q3. Exporting the schedule as hints. Precisely: E1 comes with a score, but also reveals the schedule behind it (what to fuse, how to tile the loops, and what to keep on chip), which can be used as performance engineering hints. We have added an example for attention in Appendix I and a pointer in Section 3.3.

Reviewer_YTcVOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new energy metric and modeling approach. At a high level, their approach is similar to O(N) characterizations of the algorithm or "limits studies" of the past. Their metric, E1, takes naive PyTorch code and estimates the best energy consumption of the model assuming an idealized hardware accelerator architecture. The paper presents results showing how such an approach could be very helpful across different models, algorithmic improvements, hardware and software optimizations, etc.

Critical strengths
  • I like the systematic approach to consider limits for energy
  • The experimental results and the understanding of remaining headroom across different approaches was interesting
Critical weaknesses
  • While the paper talks about energy, it primarily focuses on performance. I would be curious to see if there was a starker difference between performance and energy in their experiments
  • the paper focuses on inference and also an idealized hardware architecture. I would have liked more discussion on the sensitivity to various new radical hardware acceleration approaches.
  • while interesting as a theoretical construct, I would have liked more details on how the authors expect this metric to be used. For example, starting with a new architecture and walking through how the model helped evolve the architecture and achieve "near-oracle" energy. (I am also curious if the authors think that there will be cases when the actual energy could be better than the E1 number, and if so, under what assumptions.)
Questions for authors

See weaknesses above

1 comment
AuthorsOpenReview ↗

Response to Reviewer YTcV

Thank you for the review, and for highlighting E1's systematic approach to energy limits and the headroom it reveals across implementations.

W1. Energy versus performance. E1 is an energy metric and compares results in Joules. Still, performance optimizations often save energy too, so the two can look alike. We agree the submission did not show how energy and latency differ. We have added a paragraph in Section 4 that explains this and shows that where energy and latency diverge, E1 follows energy, backed by Appendix J (Table 5). In particular:

  • Under eager PyTorch, lengthening a Qwen3-8B prompt from 128 to 512 tokens has little effect on latency (0.98x), while energy with idle power subtracted rises 2.40x. E1 rises 2.31x.
  • Going from batch 1 to batch 8 raises latency 1.44x and idle-subtracted energy 3.98x. E1 rises 4.03x.

W2. Inference, the idealized architecture, and radical hardware. E1 scores a forward pass because inference is expected to dominate long-term AI energy use; training is outside this paper. An accelerator affects a model's energy in two fundamental ways: the cost of computation and the cost of moving data to and from memory. E1 counts exactly these two. The new Appendix H shows how the scores of three models shift as the ratio between compute and memory energy cost changes. In Section 2.2 we have added designs that do not compute this way, such as analog in-memory, neuromorphic and optical ones, as out of scope.

W3. How the metric is used. We have updated Section 4 with an example: if 30 of the 36 layers of Qwen3-8B attended only to a 1,024-token window (a choice that Gemma 4 makes), the E1 score of a 32K-token prompt would fall from 370 J to 257 J. This takes minutes on a CPU, before any optimized kernel is written.

W3. Can energy be below E1? Yes, in rare cases: when the device spends less energy per operation than the energy table assumes. We clarify this in Section 4.

Comments on the paper

1 comment
AuthorsOpenReview ↗

To all reviewers

Thank you all for your constructive reviews and for helping us improve the paper. We have addressed your comments and uploaded a revised PDF with the new text in blue. The highlights:

  • Full-model experiments at longer prompts and under optimized implementations up to vLLM (Section 3.4, Figure 7), as Reviewer VVH9 suggested.
  • A new paragraph on how energy differs from latency, and an example of how the metric can be used in AI research (Section 4), following Reviewer YTcV.
  • Three new appendices: Appendix H on how E1 varies under different compute/memory cost ratios (Reviewers YTcV and cWv7); Appendix I with an example of the tiling and fusion schedule that E1 selects for attention (Reviewer VVH9); and Appendix J with the detailed full-model results: prompt lengths up to 131K tokens and decode steps (Reviewer VVH9), the cases where energy and latency diverge (Reviewer YTcV), and mixture-of-experts implementations without merged experts (Reviewer cWv7).

Detailed responses to each reviewer follow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.