acceptodds
Under review as a conference paper at ICLR 2027

Joules-to-GDP: How Much Economic Value Do AI Systems Produce per Unit of Energy?

Abstract

An AI system can produce more economically valuable work while delivering less value for the energy it consumes. Yet existing evaluations usually assess economic value separately from the end-to-end energy required to produce it, leaving this tradeoff unclear. We therefore ask: **How much economic value does an AI system create per unit of energy consumed?** To answer this question, we formalize the economic value of an AI output on a task in two ways: (1) net dollar value, the output's dollar value minus AI-system execution cost, and (2) time saving, the expert's completion time minus the AI system's completion time. Dividing each by measured end-to-end energy then yields ***Net Value per Joule (NVJ)*** and ***Time Saving per Joule (TSJ)***, reported in USD/MJ and hours/MJ. We study 220 public GDPval tasks across 44 occupations contributing to gross domestic product (GDP), serving five open-weight models locally to measure energy directly and evaluating four accelerator types. Our analysis reveals three main findings. **First,** the occupation groups where AI systems succeed most often, deliver the greatest dollar value, or save the most time do not necessarily provide the greatest economic return per joule. For example, among the tested tasks, engineering work, such as CAD design, delivers about 10% more net dollar value per task than software development, such as building website features. Yet software development delivers about 27% more net value per joule. **Second,** GLM 5.2 delivers the highest average net dollar value among five evaluated models, at $72.27 per task using 0.430 MJ, equivalent in energy to about four hours of active laptop use. By contrast, Gemma 4 31B delivers 56% of that net dollar value using just 23% of the energy, achieving approximately 2.5 times the NVJ and 2.8 times the TSJ. **Third,** across four single-accelerator deployments, Qwen3.6-27B's NVJ varies by a factor of 4.32 and TSJ by 9.27, with B200 leading both. One and eight H100s also deliver similar net dollar value, but eight reduce completion time by 22% while using 3.24 times the energy. Together, these findings show that capability, cost, and speed alone are insufficient to identify where AI systems deliver the greatest economic return per joule. Our evaluation framework makes output value and expert time saving under an energy budget explicit objectives for improving models, accelerators, and software, while offering a new basis for assessing AI's potential economic effect on occupational work.

Primary area:
datasets and benchmarks
Keywords:
AI evaluation, AI agents, large language models, foundation models, AI benchmarks, energy, energy-efficient AI, inference efficiency, hardware accelerators, economics, economic productivity, economically valuable AI work, GDPval

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cS7JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Joules-to-GDP proposes evaluating AI agents by the economic benefit they deliver per unit of energy, rather than capability, cost, or speed alone. It introduces Net Value per Joule (NVJ) and Time Saving per Joule (TSJ), combining estimated output value, execution costs, expert time savings, and measured energy consumption. The study evaluates five open-weight models on 220 GDPval tasks spanning 44 occupations and tests four accelerator types. The results show that models and occupations producing the most valuable outputs do not necessarily provide the highest return per joule. For example, GLM 5.2 delivers the highest average net dollar value, but Gemma 4 31B achieves approximately 2.5 times higher NVJ using substantially less energy. Hardware configuration also matters: running Qwen3.6-27B on eight rather than one H100 reduces completion time by 22% but consumes 3.24 times more energy. Overall, the paper demonstrates a framework for energy-aware AI evaluation

Critical strengths

A useful framework that connects professional-task performance to measured energy consumption

  • The paper integrates judged output value, expert completion time, execution cost, and energy into NVJ and TSJ.
  • Importantly, energy accounting covers inference, tool use, and waiting, and unsuccessful attempts still incur cost and energy.

Concrete empirical evidence that cost and latency can miss substantial energy differences

  • The most compelling example is that GPT-OSS 120B consumes 68% less energy than GLM 5.2 despite nearly identical execution cost and completion time, reversing their relative efficiency when energy replaces cost or time as the denominator.
  • The paper also distinguishes total benefit from benefit per joule, acknowledging that a highly efficient model may produce less valuable work overall.
Critical weaknesses

The quantities determining the economic-efficiency rankings are insufficiently validated

  • A judge’s win-or-tie decision assigns the entire task value, while a loss receives zero, yet human validation covers only 32 comparisons from one model, with 71.9% agreement.
  • The paper does not establish whether judging disagreements materially change its value-weighted model comparisons.
  • Separately, task-level energy is allocated from shared batch measurements: changing the allocation rule changes eight NVJ ranks and nine TSJ ranks across the 13 occupational categories, including the TSJ leader.

The empirical coverage and demonstrated reproducibility are not strong enough to compensate for the primarily integrative novelty
 - The main analysis uses one run per task per configuration, with only five tasks per occupation. More importantly, deployment selection materially affects the comparisons: Qwen’s NVJ increases from approximately 123.4 to 402.2 USD/MJ when moving from eight H100s to one

  • This makes the main model ranking strongly configuration-dependent, yet comparable configuration sweeps are not provided for the other models. The manuscript also promises a future release of measurements and code rather than establishing an available, auditable artifact
Questions for authors

How robust are the economic-efficiency conclusions to judging errors? Given that wins and ties receive full task value, while human validation covers only 32 GLM comparisons, can you quantify how alternative judges or human adjudication change NVJ, TSJ, and the model rankings—not just judgment agreement?

Do the model comparisons hold under reasonably tuned serving configurations? What evidence shows that the principal model comparisons remain valid when GPU count, tensor parallelism, and concurrency are chosen appropriately for each model under comparable latency or throughput requirements, rather than using eight GPUs throughout the main comparison?

Reviewer_PPXYOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a methodology for measuring economic value from using AI as a function of energy-efficiency, and use it to answer questions around which occupation groups AI is most energy-efficient at, as well as variation across models and hardware configurations. The results show several interesting insights around how measuring for energy can lead to non-intuitive tradeoffs.

Critical strengths
  • systematically looking at economic value as a function of energy efficiency is a timely contribution and will help with the community discourse on sustainability of AI
  • the paper does a good job setting up and discussing the experiments and I enjoyed reading about the various interesting tradeoffs
Critical weaknesses

-while interesting, there is nothing really super novel about the paper. the metrics methodology is fairly straightforward and the results while interesting are relatively straightforward

  • the paper talks about expert-with-ai-assistance but I would have liked a deeper discussion on that
  • the methodology relies on an AI judge and I would have liked to have seen more depth in that discussion
  • the hardware and models discussion is fairly shallow and could have benefited from more depth - going into deeper aspects of sensitivity (for example, hardware with different memory configurations, speeds, etc) as well as providing more intuition on the results
Questions for authors
  • see comments above
  • not question per se, but it would be interesting to consider extending this to carbon footprint as well.
Reviewer_dQgdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a framework for the evaluation of the efficiency and impact of AI systems based on the returns for task-completion rather than by using the the dollar cost or time savings per unit of energy relative to an expert completing a task. The authors identify variation in AI system efficiency across variations in model size, task, and model choice.

Critical strengths
  1. The proposed framework incorporates the economic impact of an AI system whereas previous work is primarily limited to direct evaluations or analytic models of the resources required for development and deployment of these systems. The framework is extensible across AI systems, tasks, and hardware platforms.
Critical weaknesses
  1. Inconsistent Methodology.
    1. All models serve requests at batch size 20 (L261). This is not a comparable setting across models, as in practice, smaller models would either downscale the number of GPUs used or serve larger batch size -- both of which would increase the per-request efficiency.
    2. Likewise, this confounds the claims in Table 1 which state that GLM and OSS 120B have similar execution costs. If measuring the resource cost of the OSS model, it is an unfair comparison to use a baseline in which one model is achieving high hardware utilization but the other is not. Appropriate batching or downscaling would improve the resource utilization and reduce the cost. Additionally, in L375, it is stated that GLM is served at fp8, which reduces the execution cost for GLM but this is not applied to OSS.
  2. Limited Analysis
    1. The work does not provide justification as to where different models or hardware platforms differ in their task performance or energy consumption.
    2. The work provides limited insight into the variance in time or energy used by different tasks. For instance, are the tasks for certain occupation groups more or less time intensive for the evaluated AI systems. Why might this be the case (e.g. tool calls, extended reasoning traces? longer prefill? retries?)?
  3. Writing Clarity. The paper contains undefined and unclear terminology.
    1. L137. "We use the expert-only reference as primary." What is primary referring to?
    2. Nit: In the intro, the conclusions to the Q's posed is not entirely clear and the paragraphs read as a series of results.
Questions for authors

Questions

  1. L225. How is execution cost determined? Is this priced as runtime by the market rate of GPUs. That is unclear from the writing.
  2. Why was energy selected as the resource metric as opposed to other forms of environmental impact (e.g. carbon, water, public health)?
  3. L193: Please provide additional details of the IPW energy measurements, what is included in this measurement (GPU only? CPU? Facility PUE?)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.