- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Diffusion Energy Divergence (DED) for comparing real and generated distributions when a sample distance may not be of negative type. The distance defines a nearest-neighbor graph on real reference samples. Held-out real and generated samples are then placed in a shared diffusion embedding, where energy distance is computed. Experiments cover depth maps, controlled video perturbations, video generator settings, and manipulations of VBench samples.
Critical strengths
- The paper identifies a relevant limitation of distance-based distributional evaluation. Its negative-type checks and the reversed AbsRel rankings provide concrete evidence of this problem.
*The proposed solution is reasonable and well motivated. It uses local distance information and evaluates all generators in a shared reference geometry.
Critical weaknesses
-
Baseline comparisons are few and gain does not seem clear. The practical advantage over MMD baselines remains unclear. The main experiments lack a strong comparison with MMD using a valid, appropriately chosen kernel. The AbsRel experiment shows that DED can fix a failure of direct energy distance. However, this does not establish why DED should be preferred to other valid distributional methods. Video and audio comparisons also use different representations for DED and the baselines, so the added benefit of diffusion remains unclear.
-
To judge generated distributions, broad experiments should be held on image data. The image experiment in Appendix C.4 uses feature perturbations under an artificial decoder, rather than actual generated image sets evaluated against metrics such as CMMD or KID. Rankings vary substantially across real reference corpora (Table 6), and Table 12 reports negative correlations with human temporal-consistency ratings. These findings do not invalidate the construction, but they limit the evidence that its current implementation reliably evaluates generator quality across datasets.
Questions for authors
-
How does DED compare with a strong MMD baseline on the main tasks? Could the authors include a comparison using the same domain information, reference set, and sample sizes? This would help explain when practitioners should choose DED. For example, in Table 1, DED uses optical-flow profiles, whereas FVD uses S3D features. Could the authors include a valid MMD baseline using the same motion information as DED? This would help clarify what the diffusion construction adds beyond the choice of motion-sensitive features.
-
On the existing experiments, how do the energy-only, off-support-only, and combined tests compare? This would clarify which component drives the reported improvements.
-
With the evaluation samples held fixed, how much do scores and rankings change when only the reference set is resampled or drawn from another corpus? Table 6 changes both the reference and real evaluation sets.
5 comments
Response Part 2
Evaluation on real image generators. We have substantially expanded the image experiments beyond the controlled feature perturbations in the original submission. We now evaluate released samples from 13 CIFAR-10 and 11 ImageNet-256 generators, using 10,000 images per generator.
The new experiments evaluate both sensitivity to mode collapse and agreement with human judgments. DED tracks increasing mode collapse more closely than KID, energy distance, and CMMD. For example, on ImageNet-256, its Spearman correlation with mode-collapse severity is 0.66, compared with 0.02 for CMMD.
Using the human evaluation of Stein et al. (2023), we additionally obtain:
| Dataset | Spearman correlation with human rankings |
|---|---|
| CIFAR-10 (13 generators) | 0.96 |
| ImageNet-256 (11 generators) | 0.82 |
Both correlations are significantly positive, with 95% confidence intervals excluding zero. The generators span GANs, diffusion models, masked and autoregressive transformers, a VAE, and a normalizing flow. These results establish DED's practical relevance beyond synthetic perturbations and demonstrate strong agreement with human judgments across diverse real image generators
W2 / Q3. Reference sensitivity with fixed evaluation samples. Our original Table 6 examines reference sensitivity for the VBench generators but varies both the reference set and the real evaluation set. We agree that holding the evaluation samples fixed is necessary to isolate the effect of the reference geometry. Following the reviewer's suggestion, we conducted an additional experiment on depth maps, holding the evaluation sets fixed while redrawing references within and across sources (four scene types, all scenes, and a pool of the four), with 20 draws per setting.
Changing the reference source accounts for only 4% of the variation in rankings, with close agreement across sources (Kendall's τ = 0.93–0.97). Thus, in this setting, the rankings depend primarily on the real distribution against which samples are evaluated, rather than on the particular reference geometry.
Our experiments also support a simple practical recommendation: randomly sample the reference from the real data. Specialized coverage-based selection can instead reduce sensitivity to distributional changes. We have added this guidance to the revised manuscript.
W2 (continued). Negative correlations with human temporal-consistency ratings in Table 12. Our original submission shows that several VBench temporal dimensions can favor static or low-motion outputs. Repeating a real clip's first frame improves subject consistency and flickering in nearly every case (Appendix L).
DED measures discrepancies from the distribution of real motion under the chosen task-specific distance, rather than temporal consistency alone. These objectives need not agree, particularly when low-motion outputs receive high consistency ratings. Nevertheless, we acknowledge that the current motion distance does not establish DED as a general-purpose measure of perceptual temporal quality. The new image experiments provide complementary evidence that DED can agree strongly with human judgments when the underlying distance captures the relevant perceptual attributes.
This distinction is important. The diffusion construction guarantees a valid divergence between the embedded distributions, while its perceptual relevance depends on the task-specific distance used to construct the embedding. We have clarified this distinction in the revised manuscript.
Response Part 3
Q2. Energy-only, off-support-only, and combined evaluation. We now report all three variants on depth maps and point clouds. The results clarify their complementary roles:
- Energy component: Correctly ranks all tested depth degradations, including mode collapse.
- Off-support component: Detects samples outside the reference but cannot reliably rank mode collapse, since collapsed samples are copies of real samples and can remain within the reference support.
- Combined test: Detects distributional changes earlier when either component alone lacks sensitivity.
For example, at the lowest nonzero depth corruption severity, 78% of the corrupted patches fall outside the reference support. The original submission already shows that the energy component fails to detect this change (p = 0.99), whereas the combined test detects it (p = 0.01). Our expanded component-wise experiments further establish when each statistic is informative. We also evaluated several strategies for merging the components into a single ranking score, including rank sums, combined p-values, and standardized sums. All tested combinations lose the ability to correctly rank increasing mode collapse. We therefore retain the divergence and off-support rate as separate diagnostic quantities for evaluation, and use the combined permutation test for hypothesis testing. This preserves sensitivity to both distributional changes within the reference geometry and departures from its support, while controlling type-I error.
We appreciate the reviewer's suggestions, which have helped strengthen the empirical foundations of DED. The revised manuscript now compares DED against valid, tuned alternatives on matched task information, evaluates large-scale samples from real image generators, isolates reference sensitivity with fixed evaluation samples, and establishes the complementary roles of the divergence and off-support statistics.
The new results demonstrate that diffusion geometry provides benefits beyond correcting negative-type violations, particularly for structured distributions and diverse degradation types. We hope these additions resolve the reviewer's concerns and would be grateful if they would reconsider their assessment. We welcome further questions during the discussion period.
Response Part 1
We thank the reviewer for the detailed and constructive feedback, particularly for asking what DED adds over well-chosen valid alternatives and for proposing a controlled reference-sensitivity experiment. We have conducted both experiments and substantially expanded our evaluation. The new results clarify the benefits of diffusion geometry beyond statistical validity, the robustness of reference selection, and the complementary roles of the divergence and off-support statistics.
W1 / Q1. Comparison with valid, well-chosen MMD and other alternatives. We have added controlled comparisons in which competing methods receive the same task-specific distance, real data, and evaluation samples. We report the best settings of the alternatives selected retrospectively from the experiments, while keeping DED's settings unchanged.
On NYU depth maps, we evaluate three degradation types across 20 random seeds. The following table reports each method's detection rate on its hardest degradation.
DED achieves the highest worst-case detection rate without task-specific tuning. Importantly, no single MMD bandwidth matches DED across all three degradations simultaneously. This demonstrates the practical value of constructing a multiscale diffusion geometry rather than selecting a kernel bandwidth for each evaluation problem.
We also investigate whether the benefits extend beyond correcting violations of negative type:
These results indicate that diffusion geometry can improve sensitivity to distributional changes even when statistical validity is not an issue, particularly for distributions with curved or low-dimensional structure.
We have also added an analysis explaining the failure of direct energy distance. When the distance violates negative type, some spectral components contribute with the wrong sign. On depth maps, these components account for the reversed degradation rankings. In the direction corresponding to the strongest violation, direct energy distance fails to detect any change, whereas DED detects the change in up to 75% of trials.
W1 / Q1 (continued). Matched representations for video and audio. We agree that the original comparisons with FVD and audio Fréchet distances did not isolate the contribution of diffusion geometry because the representations differed.
We now include comparisons on identical task-specific inputs. For the time-series experiments, all distance-based methods receive the same motion or audio information. DTW violates negative type (Table 15, now expanded to include audio), so direct energy distance and distance-induced kernels lack the validity guarantees available in Euclidean settings.
The revised comparisons therefore distinguish the effects of representation choice and distributional comparison. We retain FVD and the audio Fréchet distances as established domain-specific benchmarks, rather than as controls that isolate the contribution of DED.