acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Generative Models via Diffusion Energy Divergence

Abstract

Generative models now produce diverse outputs, including depth maps, point clouds, video, whose quality is naturally assessed with task-specific distances between individual samples, e.g. AbsRel for depth, Chamfer distance for point clouds. Turning these distances into divergences between distributions is difficult, though, as the standard tools like energy distance and distance-induced maximum mean discrepancy (MMD) require negative type, a global condition that several commonly used distances violate on real data. We introduce the Diffusion Energy Divergence (DED), which uses a task-specific distance only to connect samples to their nearest neighbors and weight the resulting graph edges. The normalized graph Laplacian is positive semidefinite for any symmetric, nonnegative weights, and its heat semigroup therefore yields a Euclidean diffusion embedding in which energy distance is a valid divergence. This embedding depends on the original distance only through the neighbor lists and edge weights. We fit the graph once on real data and place all evaluation and generated samples on it by a Nyström extension, giving every generator the same diffusion geometry. A permutation test combines the divergence with an off-support check and has exact size under the null for any such distance. On NYU depth maps, energy distance computed directly on AbsRel ranks two depth degradations in reverse order of severity (Spearman for both), while DED ranks them correctly ( and ). On VBench, DED detects mode collapse in 16 of 20 generators, although the collapse leaves the expected VBench scores unchanged.

Primary area:
generative models
Keywords:
generative metrics, task similarities, multiple modalities, MMD, energy distance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2m6SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Diffusion Energy Divergence (DED) for comparing real and generated distributions when a sample distance may not be of negative type. The distance defines a nearest-neighbor graph on real reference samples. Held-out real and generated samples are then placed in a shared diffusion embedding, where energy distance is computed. Experiments cover depth maps, controlled video perturbations, video generator settings, and manipulations of VBench samples.

Critical strengths
  • The paper identifies a relevant limitation of distance-based distributional evaluation. Its negative-type checks and the reversed AbsRel rankings provide concrete evidence of this problem.

*The proposed solution is reasonable and well motivated. It uses local distance information and evaluates all generators in a shared reference geometry.

Critical weaknesses
  • Baseline comparisons are few and gain does not seem clear. The practical advantage over MMD baselines remains unclear. The main experiments lack a strong comparison with MMD using a valid, appropriately chosen kernel. The AbsRel experiment shows that DED can fix a failure of direct energy distance. However, this does not establish why DED should be preferred to other valid distributional methods. Video and audio comparisons also use different representations for DED and the baselines, so the added benefit of diffusion remains unclear.

  • To judge generated distributions, broad experiments should be held on image data. The image experiment in Appendix C.4 uses feature perturbations under an artificial decoder, rather than actual generated image sets evaluated against metrics such as CMMD or KID. Rankings vary substantially across real reference corpora (Table 6), and Table 12 reports negative correlations with human temporal-consistency ratings. These findings do not invalidate the construction, but they limit the evidence that its current implementation reliably evaluates generator quality across datasets.

Questions for authors
  1. How does DED compare with a strong MMD baseline on the main tasks? Could the authors include a comparison using the same domain information, reference set, and sample sizes? This would help explain when practitioners should choose DED. For example, in Table 1, DED uses optical-flow profiles, whereas FVD uses S3D features. Could the authors include a valid MMD baseline using the same motion information as DED? This would help clarify what the diffusion construction adds beyond the choice of motion-sensitive features.

  2. On the existing experiments, how do the energy-only, off-support-only, and combined tests compare? This would clarify which component drives the reported improvements.

  3. With the evaluation samples held fixed, how much do scores and rankings change when only the reference set is resampled or drawn from another corpus? Table 6 changes both the reference and real evaluation sets.

5 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for the detailed and constructive feedback, particularly for asking what DED adds over well-chosen valid alternatives and for proposing a controlled reference-sensitivity experiment. We have conducted both experiments and substantially expanded our evaluation. The new results clarify the benefits of diffusion geometry beyond statistical validity, the robustness of reference selection, and the complementary roles of the divergence and off-support statistics.

W1 / Q1. Comparison with valid, well-chosen MMD and other alternatives. We have added controlled comparisons in which competing methods receive the same task-specific distance, real data, and evaluation samples. We report the best settings of the alternatives selected retrospectively from the experiments, while keeping DED's settings unchanged.

On NYU depth maps, we evaluate three degradation types across 20 random seeds. The following table reports each method's detection rate on its hardest degradation.

MethodDetection rate
DED (ours, default settings)0.825
MMD (retrospectively tuned bandwidth)0.817
MMD (default bandwidth)0.783
Landmark MDS0.800
Affinity vectors0.792
Energy distance (valid depth distance)0.750
Energy distance (AbsRel)0.000

DED achieves the highest worst-case detection rate without task-specific tuning. Importantly, no single MMD bandwidth matches DED across all three degradations simultaneously. This demonstrates the practical value of constructing a multiscale diffusion geometry rather than selecting a kernel bandwidth for each evaluation problem.

We also investigate whether the benefits extend beyond correcting violations of negative type:

  • Real image generators: On image embeddings, where the underlying Euclidean distances are already of negative type and the competing statistics are valid, DED tracks increasing mode collapse more closely than KID, energy distance, and CMMD (see the general response).
  • Structured distributions: On a sphere, Swiss roll, ring of Gaussians, and low-dimensional subspace, DED detects distributional changes 1.5–2.7 times as frequently as the best of energy distance, Gaussian MMD, and landmark MDS.

These results indicate that diffusion geometry can improve sensitivity to distributional changes even when statistical validity is not an issue, particularly for distributions with curved or low-dimensional structure.

We have also added an analysis explaining the failure of direct energy distance. When the distance violates negative type, some spectral components contribute with the wrong sign. On depth maps, these components account for the reversed degradation rankings. In the direction corresponding to the strongest violation, direct energy distance fails to detect any change, whereas DED detects the change in up to 75% of trials.

W1 / Q1 (continued). Matched representations for video and audio. We agree that the original comparisons with FVD and audio Fréchet distances did not isolate the contribution of diffusion geometry because the representations differed.

We now include comparisons on identical task-specific inputs. For the time-series experiments, all distance-based methods receive the same motion or audio information. DTW violates negative type (Table 15, now expanded to include audio), so direct energy distance and distance-induced kernels lack the validity guarantees available in Euclidean settings.

The revised comparisons therefore distinguish the effects of representation choice and distributional comparison. We retain FVD and the audio Fréchet distances as established domain-specific benchmarks, rather than as controls that isolate the contribution of DED.

AuthorsOpenReview ↗

Response Part 2

Evaluation on real image generators. We have substantially expanded the image experiments beyond the controlled feature perturbations in the original submission. We now evaluate released samples from 13 CIFAR-10 and 11 ImageNet-256 generators, using 10,000 images per generator.

The new experiments evaluate both sensitivity to mode collapse and agreement with human judgments. DED tracks increasing mode collapse more closely than KID, energy distance, and CMMD. For example, on ImageNet-256, its Spearman correlation with mode-collapse severity is 0.66, compared with 0.02 for CMMD.

Using the human evaluation of Stein et al. (2023), we additionally obtain:

DatasetSpearman correlation with human rankings
CIFAR-10 (13 generators)0.96
ImageNet-256 (11 generators)0.82

Both correlations are significantly positive, with 95% confidence intervals excluding zero. The generators span GANs, diffusion models, masked and autoregressive transformers, a VAE, and a normalizing flow. These results establish DED's practical relevance beyond synthetic perturbations and demonstrate strong agreement with human judgments across diverse real image generators

W2 / Q3. Reference sensitivity with fixed evaluation samples. Our original Table 6 examines reference sensitivity for the VBench generators but varies both the reference set and the real evaluation set. We agree that holding the evaluation samples fixed is necessary to isolate the effect of the reference geometry. Following the reviewer's suggestion, we conducted an additional experiment on depth maps, holding the evaluation sets fixed while redrawing references within and across sources (four scene types, all scenes, and a pool of the four), with 20 draws per setting.

Changing the reference source accounts for only 4% of the variation in rankings, with close agreement across sources (Kendall's τ = 0.93–0.97). Thus, in this setting, the rankings depend primarily on the real distribution against which samples are evaluated, rather than on the particular reference geometry.

Our experiments also support a simple practical recommendation: randomly sample the reference from the real data. Specialized coverage-based selection can instead reduce sensitivity to distributional changes. We have added this guidance to the revised manuscript.

W2 (continued). Negative correlations with human temporal-consistency ratings in Table 12. Our original submission shows that several VBench temporal dimensions can favor static or low-motion outputs. Repeating a real clip's first frame improves subject consistency and flickering in nearly every case (Appendix L).

DED measures discrepancies from the distribution of real motion under the chosen task-specific distance, rather than temporal consistency alone. These objectives need not agree, particularly when low-motion outputs receive high consistency ratings. Nevertheless, we acknowledge that the current motion distance does not establish DED as a general-purpose measure of perceptual temporal quality. The new image experiments provide complementary evidence that DED can agree strongly with human judgments when the underlying distance captures the relevant perceptual attributes.

This distinction is important. The diffusion construction guarantees a valid divergence between the embedded distributions, while its perceptual relevance depends on the task-specific distance used to construct the embedding. We have clarified this distinction in the revised manuscript.

AuthorsOpenReview ↗

Response Part 3

Q2. Energy-only, off-support-only, and combined evaluation. We now report all three variants on depth maps and point clouds. The results clarify their complementary roles:

  • Energy component: Correctly ranks all tested depth degradations, including mode collapse.
  • Off-support component: Detects samples outside the reference but cannot reliably rank mode collapse, since collapsed samples are copies of real samples and can remain within the reference support.
  • Combined test: Detects distributional changes earlier when either component alone lacks sensitivity.

For example, at the lowest nonzero depth corruption severity, 78% of the corrupted patches fall outside the reference support. The original submission already shows that the energy component fails to detect this change (p = 0.99), whereas the combined test detects it (p = 0.01). Our expanded component-wise experiments further establish when each statistic is informative. We also evaluated several strategies for merging the components into a single ranking score, including rank sums, combined p-values, and standardized sums. All tested combinations lose the ability to correctly rank increasing mode collapse. We therefore retain the divergence and off-support rate as separate diagnostic quantities for evaluation, and use the combined permutation test for hypothesis testing. This preserves sensitivity to both distributional changes within the reference geometry and departures from its support, while controlling type-I error.

We appreciate the reviewer's suggestions, which have helped strengthen the empirical foundations of DED. The revised manuscript now compares DED against valid, tuned alternatives on matched task information, evaluates large-scale samples from real image generators, isolates reference sensitivity with fixed evaluation samples, and establishes the complementary roles of the divergence and off-support statistics.

The new results demonstrate that diffusion geometry provides benefits beyond correcting negative-type violations, particularly for structured distributions and diverse degradation types. We hope these additions resolve the reviewer's concerns and would be grateful if they would reconsider their assessment. We welcome further questions during the discussion period.

Reviewer_VA2TOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper "Evaluating generative models via diffusion energy divergence” is located in an important question regarding the quality of density estimation in generative models (that one might call generalization) and the “perceived” quality of the generated samples. According to the current state of research, there are different criteria that might mutually contradict (see FID vs CMMD in Jayasumana et al., 2024). The Authors find new metrics based on heat diffusion embeddings that are experimentally tested on videos and depth maps and shown to outperform state-of-the-art techniques.

Critical strengths

The paper in question bases its idea on the properties of heat kernels, going beyond the mere comparison between nearest neighbours of a graph from previous approaches.

Critical weaknesses

Generally speaking I find the main text not very fluent in readability. Moreover the motivation (for choosing the heat kernel in MMD) is not well stated both in the introduction, along the early sections and the conclusions. There is not a systematic comparison between DED and CMMD that isolates the limitations of CMMD that should be overcome.

Questions for authors

I will now present my comments and questions with the goal of improving the quality of the presented work. I will divide questions in Major and Minor issues. I ask the Authors to address my points extensively.

Major issues:

Q1: Why is it better to adopt the heat kernel instead of other characteristic kernels? What is the physical intuition behind it? According to my comprehension, the Authors need a positive-definite kernel, yet the need for searching this kernel should be shown experimentally since the beginning and it is not clear at all later from Sections 3 and 4. Contribution expressed at line 065 is not contained in the main text, and table 15 mentioned in the introduction is relegated to the Appendices, while such evidence should be shown quite early to provide motivation. In fact, still on line 065, the reader is not reconducted to any part of the text while the following items refer to propositions and experimental sections. On line 053 the diffusion embedding is mentioned, but it is not clear why.

Q2: I believe the Authors should include in the paper, even briefly and possibly attach it to figure 1, the procedure according to the which DED is measure, because it is not straightforward from Section 4 but only explained later in Section 6: consider a reference set of points, build a graph, embed them in the diffusion geometry at different diffusion times, compare embeddings according to energy distance i.e. MMD with a heat kernel. Naively, I would like to read the gist of DED in the introduction of Section 4.

Q3: Subsection 4.2 is not understandable, e.g. “(With every mode retained, every µk ̸= 0 and every sample a reference point, these means would determine the distribution.)“

Minor issues:

Q4: It is not very clear from the discussion what kind of distributions DED is confronting. Authors should include an explicit expression of DED in Section 4, such as DED(P,Q)=EX∼P,Y∼Q[D2(Φ(X),Φ(Y))]DED(P,Q) = \mathbb{E}_{X \sim P, Y \sim Q}\left[D_2(\Phi(X), \Phi(Y))\right].

Q5: I believe paragraph “Graph Construction from a Distance” should be located between paragraphs “Heat Kernels on a Manifold” and “Heat Kernels on a Sample Graph” for continuity.

Q6: Line 215: the distance between two diffusion embeddings is not clear as it is written now. Please write it more clearly in an equation environment.

5 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for the constructive feedback, particularly the concrete suggestions for improving the presentation and the request for a systematic comparison with CMMD. We have adopted all presentation suggestions and substantially expanded our experiments. The new results clarify both the practical advantages of DED over CMMD and the specific contribution of diffusion geometry, as discussed below.

W1. Systematic comparison between DED and CMMD. We have added experiments comparing DED with CMMD on real image generators. On 11 ImageNet-256 generators, we evaluate how each metric responds to increasing mode collapse. CMMD fails to track the increasing loss of modes (Spearman ρ = 0.02), whereas DED exhibits a substantially stronger correlation (ρ = 0.66). DED also outperforms CMMD in the corresponding CIFAR-10 experiments. These results demonstrate that the choice of distributional geometry can materially affect sensitivity to mode collapse, even when evaluating real generators.

We have also rewritten the opening of Section 4 to clarify the relationship between the two methods. CMMD compares CLIP embeddings using a Gaussian kernel. DED instead constructs a shared diffusion embedding from a task-specific distance, using only local relationships between real samples. The distinction is therefore not simply the choice of kernel, but how the geometry underlying the distributional comparison is constructed.

Q1: Why the heat kernel? Physical intuition and negative-type evidence. The validity of our construction does not depend on the heat kernel specifically: any nonnegative spectral function of the graph Laplacian defines a positive-semidefinite kernel. What distinguishes the heat kernel is its natural multiscale interpretation, governed by diffusion time.

Intuitively, heat released at a sample spreads along the graph edges. Two samples are close in diffusion geometry when their diffusion profiles are similar, reflecting connectivity through local paths rather than their direct pairwise distance. Since the graph retains only neighboring relationships, unreliable long-range distances do not directly determine this geometry.

Small diffusion times capture fine-scale neighborhood structure, whereas larger times capture progressively coarser relationships. Combining multiple diffusion times allows DED to account for distributional discrepancies at different geometric scales.

Our experiments support the importance of this multiscale construction. On the depth degradation experiments, using a single diffusion time yields a substantially weaker severity ranking for one degradation than using DED's three diffusion times (ρ = 0.55 vs. 0.99).

Furthermore, a new appendix compares energy distance, Gaussian MMD, KID, and Fréchet distance on the same diffusion embedding. Once the embedding is constructed, the choice among these statistics makes relatively little difference in the tested settings. This suggests that the principal contribution is the construction of the diffusion geometry, rather than the particular statistic computed on it. We have revised Section 4.2 to explain this motivation more clearly and moved the negative-type evidence from the appendix to Section 3.2, where it now directly motivates the proposed construction.

Q2: Introduce the DED procedure earlier. We agree that an operational overview should precede the mathematical derivations. Section 4 now begins with a five-step summary:

  1. Construct a weighted nearest-neighbor graph on a fixed real reference set using the task-specific distance.
  2. Compute a multiscale diffusion embedding from the graph Laplacian.
  3. Map held-out real and generated samples into the same embedding using the Nyström extension.
  4. Compute energy distance between their embedded distributions.
  5. Perform the combined permutation test, accounting for samples outside the reference support.

This overview makes the complete evaluation procedure explicit before introducing the mathematical details.

Q3: Clarify Section 4.2. We have rewritten this section and removed the confusing parenthetical statement.

The central distinction is now stated directly: MMD with the linear kernel induced by the diffusion embedding compares only the means of the embedded distributions. In contrast, energy distance compares the full distributions of the embedded samples, not merely their coordinate means.

Equivalently, energy distance corresponds, up to a constant factor, to squared MMD with a characteristic distance-induced kernel on the Euclidean diffusion coordinates. It therefore provides a valid divergence between the embedded distributions without requiring an additional kernel bandwidth.

AuthorsOpenReview ↗

Response Part 2

Q4: Which distributions does DED compare? Let PP and QQ denote the distributions of real and generated samples, respectively, and let Φ^\hat{\Phi} denote the diffusion embedding fitted on the real reference set. DED is defined as

D2(Φ^♯P, Φ^♯Q),D^2\big(\hat{\Phi}_{\sharp} P,\ \hat{\Phi}_{\sharp} Q\big),

where Φ^♯P\hat{\Phi}_{\sharp} P and Φ^♯Q\hat{\Phi}_{\sharp} Q are the distributions of real and generated samples after mapping them into the shared diffusion embedding, and D2D^2 denotes energy distance.

Thus, DED compares both distributions in the same reference-defined geometry. Importantly, this geometry is fitted independently of the evaluation and generated samples and remains fixed across all generators. We have added this explicit definition near the beginning of Section 4.

Q5: Reorder the graph construction and heat-kernel paragraphs. We have adopted the suggested organization. Section 4 now introduces heat kernels on manifolds, followed by graph construction from a task-specific distance, and then heat kernels on the resulting sample graph. This provides a more natural progression from the continuous motivation to its discrete implementation.

Q6: Express the distance between diffusion embeddings explicitly. We have added a displayed equation for the squared distance between two reference-point embeddings at diffusion time t:

\mu_k^2 e^{-t\lambda_k} \big(\psi_k(i)-\psi_k(j)\big)^2,$$ where $\lambda_k$ and $\psi_k$ are the graph spectral quantities defined in Section 4, and $\mu_k=1-\lambda_k$. The expression makes explicit how the spectral modes contribute to the distance and how diffusion time progressively attenuates higher-frequency components. It corresponds to the diffusion distance of Coifman and Lafon (2006) for our one-step heat-diffusion operator, truncated to the retained modes. We appreciate the reviewer's suggestions, which have helped us improve both the presentation and empirical validation of DED. The revised manuscript now provides an explicit procedural overview, a clearer motivation for diffusion geometry, and systematic comparisons with CMMD and alternative distributional statistics. Importantly, the new experiments demonstrate that the diffusion embedding itself contributes to improved evaluation sensitivity. We hope these results and revisions resolve the reviewer's concerns and would be grateful if the reviewer would reconsider their assessment. We welcome any further questions during the discussion period.
Reviewer_VA2TOpenReview ↗

Rebuttal

I thank the Authors for addressing all my questions and concerns.

I do not understand whether the promised updates to the manuscript have been implemented already (and uploaded here) or not. If not, as I cannot see them by downloading the .pdf (e.g. the summary in Section 4), I would ask the Authors to do it so that I can check the new version.

Reviewer_o98ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method DED to judge generative models when normally the outputs are compared using distance between generated and real data points. the author points that such energy distances work well when the distances are negative type and many existing measurement method fails on real data. such failures create false ranking of models. Thus the proposed DED method computes distance only between close neighbours, build valid distance metric using graph fitted on real data. Experiments cover depth maps, videos, audio, ViT features and the VBench video benchmark.

Critical strengths
  • The motivation is interesting and well explained.
  • The negative type distance is tested on many distances and they prove the failure observation.
  • Building the graph on real data, so all models are judged on the same scale, is a practical and well-motivated choice for benchmarks.
  • Theoretical explanation on validity of estimation of neighbourhood distances only is sufficient and interesting.
  • Limitations is discussed with detail for better understanding.
Critical weaknesses
  • changing the real dataset type can drastically change the agreement and the paper has mentioned it and this can be termed as one of the limits of the proposed technique. However, the authors did not properly address this on how tp select a proper set for understanding the generative model capabilities.
  • DED use alone is not able to understand sample far from the real data distribution, it needs a separate check. therefore answer on the ranking is based on the combined test, not a single number. This makes it harder to use for ranking models.
  • The authors list merging the two as future work but this should be discussed in the paper itself, which sort of merging leads to how much change in ranking, the sensitivity analysis.
  • Experiments are small and uses some limited or few number of data points for videos and depths.
  • Chamfer distance on point clouds and RMSD on molecules are used to motivate the problem, but DED is not evaluated on them. most damage is reported on syntheitic changes, how often such error appears in modelling and generation? can an experiment be based on that?
  • I think the method is costly: The method need to compute the distances between all pairs in the set, and the nearest samples. With expensive distances like DTW, this could be slow for larger sets
Questions for authors
  • experiment on larger set need for better clarity
  • cloud point computation of DED is not reported in the paper
  • Can you show at least one real setting where DED agrees with human judgment
  • running time and storage needs to be reported
  • since the proposed idea is dependent on the distribution of data, how much DED changes for models when a different distribution type is used: gaussian, spherical etc.
2 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for recognizing the motivation, theoretical validity, and practical advantages of fitting a shared evaluation geometry on real data. In response to the reviewer's concerns, we have substantially expanded our experiments and revised the manuscript. Our new results demonstrate strong agreement with human judgments on 24 real image generators, evaluation at substantially larger scales, and the practical impact of negative-type violations on real generator rankings. This is performed using the publicly available data from (Stein et al. 2023).

Stein, George, et al. "Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models." Advances in Neural Information Processing Systems 36 (2023): 3732-3784.

W1. Reference-set dependence and guidance on reference selection. Our original submission (Table 6, Appendix I) examines reference sensitivity for the VBench generators, showing relatively stable rankings within the same corpus but larger changes across corpora. However, that experiment varies both the reference set constructing the diffusion geometry and the evaluation set defining the target distribution. To isolate these effects, we conducted an additional experiment on depth maps, holding the evaluation sets fixed while resampling the reference within and across different scene types, using 20 draws per setting. Changing the reference source accounts for only 4% of the variation in rankings, and rankings obtained using different reference sources agree closely (Kendall's τ = 0.93–0.97). Thus, in this setting, rankings depend primarily on the target real distribution against which generated samples are evaluated, rather than the particular reference geometry used (see our response to Reviewer 2m6S, W2 and Q3).

Our experiments support a simple rule: randomly sample the reference from the real data, pooling across sources when appropriate. Specialized coverage-based selection can instead reduce sensitivity to distributional changes. The off-support statistic already introduced in Section 5.1 provides a diagnostic of reference coverage. We have added this practical guidance to the revised manuscript.

W2-W3. Combining DED and the off-support statistic into a single ranking score. We evaluated several merging strategies, including rank sums, combined p-values, and standardized sums. All tested strategies lose the ability to correctly rank increasing mode collapse. As collapse increases, the diffusion energy component increases, while the off-support rate can decrease. Combining them can therefore obscure an important distributional failure. These results highlight the distinction between two objectives:

  • Hypothesis testing: Our combined permutation test provides a principled single decision on whether the distributions differ, with controlled type-I error (Proposition 3).
  • Model ranking: We recommend reporting the diffusion energy component and the off-support rate separately, preserving information about distributional discrepancies and samples outside the reference support.

We have added the merging experiments to the revised manuscript, providing empirical justification for reporting the two quantities separately rather than imposing an arbitrary composite score.

W4/Q1. Experimental scale. Our new image experiments use 10,000 generated images per model and up to 10,000 real reference images, covering 13 generators on CIFAR-10 and 11 on ImageNet-256.

Importantly, on CIFAR-10, agreement with human judgments remains stable across reference sizes from 1,250 to 10,000 images, indicating that a substantially smaller reference can already provide reliable rankings. We have also repeated the depth experiments across 20 random seeds to establish the robustness of our findings to sampling variability.

W5/Q2. Evaluation with Chamfer distance on point clouds. We have added a point-cloud experiment that confirms that Chamfer distance violates negative type on real point clouds. Under rotations of increasing magnitude, direct energy distance on Chamfer ranks the rotations in reverse order of severity, whereas DED recovers the correct ordering.

This extends our original findings on AbsRel to another widely used geometric distance. Although we have not added a corresponding molecular experiment with RMSD, the point-cloud results provide further empirical validation of our central motivation.

AuthorsOpenReview ↗

Response Part 2

W5 (continued). Do negative-type violations affect rankings of real generators? The violations reported in Table 15 are measured on real data, not artificially constructed distances. All examined distances except SILog and exact optimal transport fail the negative-type test on the sampled real data. We also examine whether these violations have practical consequences for evaluating actual generators, without applying synthetic degradations. On the 20 VBench generators, DED and direct energy distance, both constructed from the same DTW distance, disagree on the ordering of approximately half of all generator pairs.

While disagreement alone does not establish which ranking better reflects human preferences, it demonstrates that the choice of distributional comparison materially affects real benchmark rankings. Our controlled degradation experiments establish that direct energy distance can reverse known severity orderings, while the new human evaluation below provides independent evidence of DED's practical usefulness.

W6 / Q4. Computational cost, runtime, and storage. DED amortizes the cost of expensive task-specific distances across generators by fitting a single reusable reference graph. Let r denote the reference size, n the number of generated samples, and CdC_d the cost of evaluating the original distance.

  • Reference construction: O(r2Cd)O(r^2C_d) computed once and reused for all generators..
  • Per-generator distance computation: O(nrCd)O(nrC_d), so the number of expensive distance evaluations grows linearly with the number of generated samples for a fixed reference.
  • Distributional comparison: Exact energy-distance computation remains quadratic in the evaluation sample size, but operates on inexpensive, low-dimensional Euclidean embeddings rather than the original task-specific distance.

In contrast, direct energy distance and full pairwise kernel comparisons constructed from the original distance require quadratic numbers of expensive distance evaluations. Large reference sets are also not always necessary. On CIFAR-10, DED achieves essentially the same agreement with human rankings using 1,250 or 10,000 reference images, while our depth experiments use only 300 reference samples. As a concrete example, computing all DTW distances for a reference graph of 1,000 video clips takes 15 seconds on a CPU. We have added a table reporting computational costs, runtime, and storage requirements. Finally, DED's permutation test controls type-I error at any sample size. Our depth, video, and VBench experiments demonstrate useful sensitivity with 100–150 samples per evaluation set, while the new image experiments demonstrate scalability to substantially larger evaluations.

Q3. Agreement with human judgments on real generative models. We have added a new evaluation using the human evaluation setup of Stein et al. (2023), in which participants judge whether generated images are real. DED exhibits strong agreement with the resulting generator rankings:

Datasetρ\rho with human rankings
CIFAR-10 (13 generators)0.96
ImageNet-256 (11 generators)0.82

Both correlations are significantly positive, with 95% confidence intervals excluding zero. The evaluated generators span GANs, diffusion models, masked and autoregressive transformers, a VAE, and a normalizing flow. Agreement on CIFAR-10 remains strong across all tested reference sizes, from 1,250 to 10,000 real images. These results demonstrate DED's ability to recover human-aligned rankings of actual generators across diverse model families and image datasets.

Q5. Sensitivity to different underlying distributions. We have expanded our synthetic experiments to include Gaussian and spherical distributions, a ring of Gaussians, a Swiss roll, and a low-dimensional subspace, covering multimodality, curved manifolds, and low intrinsic dimensionality. We evaluate Euclidean distance and two distances that violate negative type, comparing DED against direct energy distance, Gaussian MMD, and landmark MDS. Across all tested distribution types, DED detects distributional changes more frequently than the strongest baseline, while maintaining an empirical false-positive rate below 5%.Thus, its benefits are not restricted to a particular distribution type or to distances that violate negative type

We appreciate the reviewer's suggestions, which have helped us strengthen the practical validation of our method. The new experiments directly address the principal concerns about reference sensitivity, computational cost, experimental scale, and applicability to real generators. In particular, they demonstrate strong agreement with human judgments and establish that the failures motivating DED have meaningful consequences for real benchmark rankings. We hope these additions resolve the reviewer's concerns and would be grateful if they would reconsider their assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.