- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Joules-to-GDP proposes evaluating AI agents by the economic benefit they deliver per unit of energy, rather than capability, cost, or speed alone. It introduces Net Value per Joule (NVJ) and Time Saving per Joule (TSJ), combining estimated output value, execution costs, expert time savings, and measured energy consumption. The study evaluates five open-weight models on 220 GDPval tasks spanning 44 occupations and tests four accelerator types. The results show that models and occupations producing the most valuable outputs do not necessarily provide the highest return per joule. For example, GLM 5.2 delivers the highest average net dollar value, but Gemma 4 31B achieves approximately 2.5 times higher NVJ using substantially less energy. Hardware configuration also matters: running Qwen3.6-27B on eight rather than one H100 reduces completion time by 22% but consumes 3.24 times more energy. Overall, the paper demonstrates a framework for energy-aware AI evaluation
Critical strengths
A useful framework that connects professional-task performance to measured energy consumption
- The paper integrates judged output value, expert completion time, execution cost, and energy into NVJ and TSJ.
- Importantly, energy accounting covers inference, tool use, and waiting, and unsuccessful attempts still incur cost and energy.
Concrete empirical evidence that cost and latency can miss substantial energy differences
- The most compelling example is that GPT-OSS 120B consumes 68% less energy than GLM 5.2 despite nearly identical execution cost and completion time, reversing their relative efficiency when energy replaces cost or time as the denominator.
- The paper also distinguishes total benefit from benefit per joule, acknowledging that a highly efficient model may produce less valuable work overall.
Critical weaknesses
The quantities determining the economic-efficiency rankings are insufficiently validated
- A judge’s win-or-tie decision assigns the entire task value, while a loss receives zero, yet human validation covers only 32 comparisons from one model, with 71.9% agreement.
- The paper does not establish whether judging disagreements materially change its value-weighted model comparisons.
- Separately, task-level energy is allocated from shared batch measurements: changing the allocation rule changes eight NVJ ranks and nine TSJ ranks across the 13 occupational categories, including the TSJ leader.
The empirical coverage and demonstrated reproducibility are not strong enough to compensate for the primarily integrative novelty - The main analysis uses one run per task per configuration, with only five tasks per occupation. More importantly, deployment selection materially affects the comparisons: Qwen’s NVJ increases from approximately 123.4 to 402.2 USD/MJ when moving from eight H100s to one
- This makes the main model ranking strongly configuration-dependent, yet comparable configuration sweeps are not provided for the other models. The manuscript also promises a future release of measurements and code rather than establishing an available, auditable artifact
Questions for authors
How robust are the economic-efficiency conclusions to judging errors? Given that wins and ties receive full task value, while human validation covers only 32 GLM comparisons, can you quantify how alternative judges or human adjudication change NVJ, TSJ, and the model rankings—not just judgment agreement?
Do the model comparisons hold under reasonably tuned serving configurations? What evidence shows that the principal model comparisons remain valid when GPU count, tensor parallelism, and concurrency are chosen appropriately for each model under comparable latency or throughput requirements, rather than using eight GPUs throughout the main comparison?