- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces E1, a metric that estimates the energy an optimized implementation of a model could achieve, computed directly from unoptimized PyTorch code with no GPU and no kernel tuning. The pipeline lowers a model to an operator graph, then an einsum graph, then searches the tiling and fusion mapspace of an abstract accelerator for the schedule that minimizes total energy against a small calibrated energy table. The paper validates E1 against energy measured on an NVIDIA H100, the same GPU the table was calibrated on, across microbenchmarks spanning three orders of magnitude, KernelBench tasks, production kernels for LLM serving across competing implementations and across five real historical versions of the same kernel, and full forward passes of a mixture-of-experts model, two dense models, and a recurrent block.
Critical strengths
- Section 3.4's mixture-of-experts test confirms on real hardware that an "8x" board energy gap between two implementations is entirely idle power, with energies agreeing within 3% once idle power is subtracted while E1 stays unchanged. Table 2's five real historical versions of FlashInfer's kernel give the same kind of externally verifiable result, with measured energy moving "-6% to +10%" across two years of development while E1's score does not move.
Critical weaknesses
-
Section 2.2 motivates E1 as "infrastructure-agnostic" across GPUs, TPUs, and RDUs, but every validation experiment measures against the same H100 used to calibrate the energy table. The hard part of that claim, generalizing across hardware families instead of implementations on one GPU, is asserted but not tested.
-
Section 2.4 excludes launch overhead entirely because "launch overhead disappears with fusion or CUDA Graphs," supported by one worked example (Section 3.4) where the underlying computation can be merged into a single grouped kernel. This does not establish the exclusion for architectures where each expert's or branch's computation cannot be grouped at all, which is exactly the kind of architecture comparison E1 is meant to support.
Questions for authors
-
Has E1 been checked against measured energy on any accelerator other than the H100 the energy table was calibrated on, and would the same tracking hold if a different GPU generation or a non-GPU target were calibrated instead?
-
Section 2.4 excludes launch overhead because it disappears with fusion or CUDA Graphs. Does that hold for architectures where each unit's computation cannot be merged into one grouped kernel, such as routing to experts of heterogeneous shape, where many small launches may be an architecturally forced cost instead of an artifact of immature kernel engineering?
-
The abstract states optimized kernels land "2-4x" above E1, narrower than the 2.3-6x range reported elsewhere in the paper (Appendix C, Figure 5). Could the abstract's range be reconciled with the fuller spread the paper itself reports?
-
Section 2.2 names SRAM-only designs as explicitly out of scope because they lack an off-chip memory boundary. Does the same limitation apply to neuromorphic or other non-von-Neumann accelerators, and should the claim of infrastructure agnosticism name that exclusion too?
Response to Reviewer cWv7
Thank you for the review, and for highlighting that E1's score does not move while the implementations of the same model or kernel change.
W1/Q1. "Infrastructure-agnostic" and other accelerators. We have added a clarification in Section 2.2 of what makes E1 infrastructure-agnostic, as its hardware target does not model a microarchitecture like other tools do. However, to meaningfully compare a score with real measurements we need reference numbers. We took those from the H100, an accelerator available to us and widely deployed. Calibrating another GPU generation or a non-GPU target would replace the reference numbers and leave the rest of E1 unchanged. We have added Appendix H to show how E1 scores shift as the ratio between compute and memory cost changes. Still, calibration is optional: one can also set these numbers directly, for example to weight memory accesses more heavily.
W2/Q2. Launch overhead when experts cannot be merged. E1 focuses on the efficiency of the model. Over time, systems optimizations have removed overheads that are not tied to it: CUDA graphs replay a launch sequence without the host, grouped GEMM kernels (e.g., in CUTLASS) run matrix products of different shapes in one launch, and megakernels such as MegaMoE run a whole mixture-of-experts layer as a single kernel. We have added an experiment on Qwen3-30B-A3B without merging any experts (Section 3.4, Figure 7c; Appendix J): with one kernel sequence per expert, the routes of the prompt fixed, and the launches replayed as a CUDA graph, board energy drops from 123 J to 20.1 J, against 14.1 J for the grouped kernel.
Q3. 2-4x in the abstract. Thank you for noticing this inconsistency. We have corrected the abstract to 1.9-6x (the added full-model experiments reach 1.9x; see Section 3.4, Appendix J).
Q4. Non-von-Neumann designs. Thank you for the suggestion. We have added analog in-memory, neuromorphic, optical and other non-von-Neumann designs to Section 2.2 as outside the scope, next to the SRAM-only designs.