- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes GQD, a weighted distance to an ideal point of zero energy and best quality, and benchmarks 140 inference configurations of text-to-image and text-to-video models, plus GPU power capping. It reports up to 81% energy savings at similar quality.
Critical strengths
The image measurements appear reproducible (Table 1 traces to the released CSV), and the practical tips may be useful. Including embodied impacts (ADP, GWP, water) goes beyond most energy-only benchmarks.
Critical weaknesses
- GQD is a weighted Euclidean distance to an ideal point (Sec. 3.3, Eq. 5, p. 5, l. 217–219), and the properties proved in App. B.2 (p. 18, l. 957) follow from that construction rather than from anything specific to energy and quality. The one design decision with empirical content is where the reference limits come from, and App. B.1 (p. 17, l. 913–917) sets them by a grid search that equalizes the score's rank correlation with its two axes on the same configurations that are then ranked. The claim that quality and energy "contribute comparably" is therefore a property of the fit, not a finding about the benchmark, and the paper's advice to freeze the limits is an acknowledgement that the population dependence it set out to avoid has been relocated into a calibration step rather than removed. Two consequences follow. Rankings under a weighted distance are governed by a single effective weight, so the score adds little beyond reading the Pareto front of Fig. 1. And because the paper also lets the weight on the impact block vary, the recommended operating point moves with that choice (Fig. 7, 14, p. 8, l. 397–415), so what is presented as a metric is really a family of rankings parameterized by a decision the paper leaves to whoever applies it.
- The evidence is descriptive rather than validating, and the uncertainty relevant to the paper's advice is not reported. Energy is treated as exact, with no repeated runs or intervals, although the recommendations are stated as energy differences between settings. Video quality rests on very few samples per dimension under a single seed, aggregated by a simple mean over raw dimensions rather than by the official protocol, so step-level and model-level video rankings are hard to distinguish from sampling noise and the totals are not comparable with published VBench numbers; Table 2 and the "133× higher" comparison (p. 7, l. 333–338) inherit that weakness. On the image side, the step sweeps change guidance along with the number of steps, so part of what is attributed to the denoising budget belongs to another variable. Finally, the headline savings are stated for two hand-picked configurations rather than as a distribution over the benchmark, and "similar quality" is a tolerance that is asserted rather than justified against human perception.
- The paper criticizes Quality per Joule for being dominated by one axis (App. C, p. 19, l. 1020–1025) but does not rule out the same failure for GQD in most of its settings. The symmetry diagnostic Δρ that would answer this is reported for a single image benchmark under one weighting (App. B.1, p. 18, l. 928–931), while the video, resolution and power-cap studies present several-fold to two-orders-of-magnitude spread along the energy axis with narrow quality ranges. In that regime a weighted distance is close to an energy ranking, and the reader cannot tell from the reported tables whether the ranking reflects quality at all. A related issue is that comparisons mix architectural and configuration effects: differences between models of different size and resolution are scored on the same axis as differences between step counts of one model, so the practical recommendation to reduce steps is entangled with model selection, and the configuration-level advice itself (fewer steps for distilled models) is largely known.
Questions for authors
- Sec. 4.2 and Table 2 (p. 7, l. 326–338): what are the video results under the official VBench protocol (full prompt suite, multiple samples per prompt, official aggregation)?
- Sec. 3.3 and Sec. 5 (p. 5, l. 208–223; p. 8, l. 401–415): with the reference limits held fixed, how much does the ranking change as the impact weight γ varies, and which configurations are stable across that range? A practitioner needs to know whether the recommended operating point is a property of the models or of the chosen weighting.