acceptodds
Under review as a conference paper at ICLR 2027

Green Quality Distance: Characterizing Energy and Quality Trade-Off in Open Text- to-Image and Text-to-Video Models

Abstract

Advances in generative technologies have led to the development of models capable of creating high-quality images and videos from text. In turn, the environmental impact of deployment and usage is a growing concern in research and industry. These models are commonly evaluated using qualitative metrics related to perceptual quality and prompt adherence. Recent research focused on measuring the consumed energy or the emitted during training and inference. The notion of Intelligence per watt (Saad-Falcon et al., 2026) enables a rating of Large Language Models based on their quality (e.g. intelligence) relative to the required power (e.g. watts). This metric can however not be applied readily to image and video models. We propose a novel formulation allowing to assess both the visual quality and environmental impact of generative image and video models for a given inference configuration: the Green Quality Distance. Our metric emphasizes the environmental impact of models by gathering data on energy consumption, abiotic resource usage, global warming potential, and water use. By benchmarking configurations (e.g. diffusion steps, GPU power cap) of open-source state-of-the-art models, we show that the right configuration reaches a given quality bar at a fraction of the energy cost, exploiting the full potential of existing models while reducing resource waste.

Primary area:
interpretability and explainable AI
Keywords:
generative model evaluation, energy efficiency, environmental impact, green ai, text-to-image generation, text-to-video generation, diffusion models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JUZWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes GQD, a weighted distance to an ideal point of zero energy and best quality, and benchmarks 140 inference configurations of text-to-image and text-to-video models, plus GPU power capping. It reports up to 81% energy savings at similar quality.

Critical strengths

The image measurements appear reproducible (Table 1 traces to the released CSV), and the practical tips may be useful. Including embodied impacts (ADP, GWP, water) goes beyond most energy-only benchmarks.

Critical weaknesses
  1. GQD is a weighted Euclidean distance to an ideal point (Sec. 3.3, Eq. 5, p. 5, l. 217–219), and the properties proved in App. B.2 (p. 18, l. 957) follow from that construction rather than from anything specific to energy and quality. The one design decision with empirical content is where the reference limits come from, and App. B.1 (p. 17, l. 913–917) sets them by a grid search that equalizes the score's rank correlation with its two axes on the same configurations that are then ranked. The claim that quality and energy "contribute comparably" is therefore a property of the fit, not a finding about the benchmark, and the paper's advice to freeze the limits is an acknowledgement that the population dependence it set out to avoid has been relocated into a calibration step rather than removed. Two consequences follow. Rankings under a weighted distance are governed by a single effective weight, so the score adds little beyond reading the Pareto front of Fig. 1. And because the paper also lets the weight on the impact block vary, the recommended operating point moves with that choice (Fig. 7, 14, p. 8, l. 397–415), so what is presented as a metric is really a family of rankings parameterized by a decision the paper leaves to whoever applies it.
  2. The evidence is descriptive rather than validating, and the uncertainty relevant to the paper's advice is not reported. Energy is treated as exact, with no repeated runs or intervals, although the recommendations are stated as energy differences between settings. Video quality rests on very few samples per dimension under a single seed, aggregated by a simple mean over raw dimensions rather than by the official protocol, so step-level and model-level video rankings are hard to distinguish from sampling noise and the totals are not comparable with published VBench numbers; Table 2 and the "133× higher" comparison (p. 7, l. 333–338) inherit that weakness. On the image side, the step sweeps change guidance along with the number of steps, so part of what is attributed to the denoising budget belongs to another variable. Finally, the headline savings are stated for two hand-picked configurations rather than as a distribution over the benchmark, and "similar quality" is a tolerance that is asserted rather than justified against human perception.
  3. The paper criticizes Quality per Joule for being dominated by one axis (App. C, p. 19, l. 1020–1025) but does not rule out the same failure for GQD in most of its settings. The symmetry diagnostic Δρ that would answer this is reported for a single image benchmark under one weighting (App. B.1, p. 18, l. 928–931), while the video, resolution and power-cap studies present several-fold to two-orders-of-magnitude spread along the energy axis with narrow quality ranges. In that regime a weighted distance is close to an energy ranking, and the reader cannot tell from the reported tables whether the ranking reflects quality at all. A related issue is that comparisons mix architectural and configuration effects: differences between models of different size and resolution are scored on the same axis as differences between step counts of one model, so the practical recommendation to reduce steps is entangled with model selection, and the configuration-level advice itself (fewer steps for distilled models) is largely known.
Questions for authors
  1. Sec. 4.2 and Table 2 (p. 7, l. 326–338): what are the video results under the official VBench protocol (full prompt suite, multiple samples per prompt, official aggregation)?
  2. Sec. 3.3 and Sec. 5 (p. 5, l. 208–223; p. 8, l. 401–415): with the reference limits held fixed, how much does the ranking change as the impact weight γ varies, and which configurations are stable across that range? A practitioner needs to know whether the recommended operating point is a property of the models or of the chosen weighting.
Reviewer_SBkaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Green Quality Distance, a metric that jointly ranks image and video generation configurations by their distance from an ideal of maximum quality and zero energy consumption

The authors benchmark 18 open-weight models across 140 configurations, varying denoising steps, output resolution, and GPU power caps.They evaluate quality using FID, CLIPScore, GenEval, and VBench, while measuring energy by integrating GPU power readings during inference.

Results show that selected configurations can reduce energy consumption by up to 81% while maintaining similar benchmark quality, demonstrating the importance of evaluating inference settings rather than model names alone.

Critical strengths

Actionable empirical evidence that inference configurations matter substantially

  • The paper evaluates 18 open-weight image and video models across 140 configurations and identifies selected settings that reduce GPU energy consumption by up to 81% while retaining similar automatic benchmark scores
  • This moves beyond ranking model names to examining the operating choices available to users

A useful distinction between energy efficiency and broader environmental trade-offs:

  • The power-capping analysis considers latency and estimated environmental impacts rather than treating reduced energy as an unqualified improvement
Critical weaknesses

Modest methodological novelty, with insufficient evidence that GQD improves configuration selection

  • GQD is a weighted Euclidean distance over normalized gaps; the appendix explicitly shows that its ranking is equivalent to a weighted sum of squared gaps
  • The comparison with quality-per-joule illustrates a weakness of that ratio, but does not establish an advantage over alternatives such as selecting the lowest-energy configuration satisfying an explicit quality requirement
  • This matters to my recommendation because the demonstrated savings establish the value of configuration profiling, not necessarily the proposed metric
  • Consequently, the experiments and their reusability must carry most of the contribution

The empirical validation and reporting are not yet strong enough to carry that contribution:

  • Although model coverage is respectable, the main benchmark tables provide point estimates without uncertainty intervals, while the central quality-preservation claim treats differences “within 10%” as similar perceptual and compositional quality.
  • This leaves insufficient evidence that the recommended savings preserve a justified, practically acceptable quality level.
  • the manuscript also leaves important measurement details unspecified, including GPU counts per configuration, batch sizes, and NVML sampling frequency, limiting independent assessment of reproducibility. The A100 GPU used was hidden in a braket in the text somewhere.
  • This is the main reason for my assessment: for a primarily empirical contribution, confidence in quality preservation and reproducibility is central, rather than a secondary reporting concern
Questions for authors

What practical advantage does GQD provide over choosing the lowest-energy configuration that satisfies an explicit quality requirement?

Are the cross-model video comparisons based on equivalent generation workloads? In particular, for the reported 133-fold energy difference, what are the exact output dimensions, frame counts, and frame rates of the compared configurations, and how much of the difference remains after accounting for any workload differences?

Can you identify where the supplementary archive documents the complete setup and measurements needed to reproduce the main results? Specifically, does it provide GPU counts per configuration, batch sizes, numerical precision, software/checkpoint versions, NVML sampling frequency and measurement boundaries, and per-run quality and energy results with scripts to regenerate the principal tables?

Reviewer_joUMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper investigates how to measure the effectiveness of an image/video generation configuration, considering both the quality of the generation and resource use.

The paper proposes a new metric, Green Quality Distance (GQD). The metric consider four configuration dimensions: model, output resolution, denoising steps, and guidance setting. Denote the configuration by cc, the metric is GQD(c)=κdQ(c)2+ωdE(c)2GQD(c) = \sqrt{\kappa d_Q(c) ^ 2 + \omega d_E(c) ^ 2} where κ+ω=1 and κ,ω≥0\kappa + \omega = 1 \text{ and } \kappa, \omega \geq 0.

Tha paper, using this new metric, benchmarks 48 configurations of open-weight text-to-image pipelines and 42 configurations of open-weight text-to-video pipelines. Results show that when considering the energy cost of the configurations, their rankings change compared to just considering the quality of the generation.

The paper later studies the trade-off between latency and energy when imposing a power cap on the GPUs. Results suggest on average, capping GPU power at 60% achieves the best balance between energy use and latency.

The paper also adapted the metric to language models. The paper finds that existing metric, intelligence (quality) per joule, is biased towards energy. When using GQD, the bias is reduced.

Critical strengths
  1. This paper evaluates the model with its configurations and provides strong evidence of the configuration-energy trade-off.
  2. The paper studied a broad range of models and benchmarks.
Critical weaknesses
  1. Practical guidance for the community on choosing weights is limited. It is unclear how users can use the method.
  2. The novelty of the method is limited. It is a weighted combination of several normalized criteria.
  3. The model-agnostic methodology is not clearly presented in the paper. The details of the experiments should be clarified more clearly.
Questions for authors
  1. Do you have a principled method for choosing the weights of each component?
  2. What additional aspects can GDQ provide over Pareto analysis?
  3. What is the model-agnostic methodology mentioned in the third contribution?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.