acceptodds
Under review as a conference paper at ICLR 2027

What FID Hides: Detecting, Ranking, and Diagnosing Deviations in Generative Evaluation

Abstract

Generative models are commonly ranked by Fréchet Inception Distance (FID) and Kernel Inception Distance (KID), yet FID's first-two-moment summary can miss distributional differences, and a reported scalar gap alone is not a calibrated test against sampling variation. FID's moment restriction has concrete consequences: on ImageNet, visually unrecognizable images optimized only to match the reference mean and covariance in Inception feature space obtain FID 24.7 versus 58.6 for held-out real images (lower is better). Moreover, FID and KID are scalar discrepancies that are unchanged when the two samples are exchanged and therefore do not encode the direction of a dispersion change: under-dispersion, as can occur in mode collapse, versus over-dispersion. We introduce ZID (Z-resolved Integrated Diagnostic), which combines six standardized components from a rank-based graph test and Gaussian-kernel tests at two bandwidths. Rather than asking one scalar to serve incompatible roles, ZID reports three linked outputs: a score for ranking candidate models, a permutation -value for testing distributional equality, and a signed dispersion readout for diagnosis. In a 19-method comparison at a fixed dimension and sample size, ZID is the only method with detection power of approximately 0.70 or higher on each of six departures. Its score is positively associated with increasing severity across all six ranking sweeps, including cases in which FID is flat or reversed. On DiT-XL/2 and SiT-XL/2 guidance sweeps, ZID's score tracks departure from real data, and its signed readout distinguishes low-guidance over-dispersion from high-guidance under-dispersion.

Primary area:
generative models
Keywords:
generative model evaluation, evaluation metrics, two-sample testing, model ranking, distributional diagnostics, permutation testing, mode collapse, Fréchet Inception Distance (FID)

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Bao4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper points out two problems of FID, which emerge when comparing only the first and second moments of real and fake feature distributions: (1) the inability to accurately distinguish between real and generated distributions, and (2) the failure to provide an analysis of aspects such as dispersion due to the reliance on a single evaluation value. To address these problems, the paper proposes a new metric called ZID (Z-resolved Integrated Diagnostic).

Critical strengths

The problems with FID highlighted in this paper are indeed valid. A compelling example is shown in Fig. 1, where unrealistic noise images yield a better FID score against real images than the FID score between two distinct sets of real images. The proposed evaluation method (ZID) serves as a convincing solution to the two identified problems, (1) the inability to accurately distinguish between real and generated distributions and (2) the failure to analyze aspects such as dispersion, despite the study's inherent limitations regarding low-dimensional embeddings and small sample sizes.

Critical weaknesses

CW1. The ZID score defined in this paper does not provide a consistent interpretation of differences between distributions; the meaning of the difference can vary with the experimental setup, creating ambiguity or difficulty for users attempting to interpret the score. For example, the ZID score takes the minimum over values 6r_(i)/i (L215-219), so the final score is determined by a single arm’s value. Among the six r_k values, three are based on Z_W and the other three on Z_D. Therefore, depending on whether the final score is determined by Z_W or Z_D, the score may have different interpretations, making the meaning of the score difficult to interpret consistently.

CW2. It appears to require further validation regarding experimental fairness. First, the FID and KID scores are typically calculated using sample sizes of 10k to 50k, which are the metrics that ZID score is primarily compared. However, the sample sizes used for comparison are substantially smaller (e.g., n = 500) than those in the originally proposed setting, which could significantly affect the reported results. Second, regarding the use of precision and recall to measure fidelity and diversity (in Section 5), these metrics are known to be inherently vulnerable to outliers [1], so they may not provide reliable indicators in regimes such as high CFG, where the number of off-manifold samples increases. Thus, it remains questionable whether the paper’s claim would hold consistent when these fairness problems are resolved.

  • [1] Naeem, Muhammad Ferjad, et al. "Reliable fidelity and diversity metrics for generative models." International conference on machine learning. PMLR, 2020.
Questions for authors

Q1. Does the ranking of existing generative models based on the FID score change when using the ZID score, and how can such changes in ranking be interpreted in the context of the method proposed in this paper?

Q2. Could the authors add statements explaining the design rationale in the sections introducing the proposed method, such as Section 3.3? Currently, the reader has to infer the author’s design intentions, which makes it difficult to assess the validity of the proposed method.

9 comments
AuthorsOpenReview ↗

Response to Bao4 (1/4): Score interpretation and detection at 10k

Thank you for your constructive comments. We clarify the interpretation and design rationale of ZID, compare results at 500, 10k, and 50k, examine how metric choice affects model selection, and validate dispersion findings without relying on Precision/Recall.

CW1. Interpreting ZID when different components dominate

Thank you for this very good point. In practice, distributions can differ in many ways, and we usually do not know the type of change in advance. The goal of ZID is to cover as many of these changes as possible. This broad coverage is our main focus at the current stage and motivates an omnibus approach.

For interpretation, the values of the six arms may provide useful hints. Each arm is standardized relative to its permutation-null variation and mapped to the same two-sided normal-reference tail scale before combination. Thus, all comparisons use the same scoring rule, although different arms can explain the response in different settings. The dispersion diagnostic is one of our attempts in this direction, supported by the controlled experiments in the paper and the human-evaluation results discussed below under CW2. Developing further interpretations of the component values is an important next step.

Regarding the Simes combination, it can respond to either one particularly strong component or several moderately strong components. For example, reference-tail values (.001,1,1,1,1,1)(.001,1,1,1,1,1) give qrank=.006q_{\mathrm{rank}}=.006, with the minimum attained at i=1i=1. Values (.003,.003,.003,1,1,1)(.003,.003,.003,1,1,1) give the same value, now with the minimum attained at i=3i=3. Thus, the minimum need not represent an isolated single-arm response. These reference-tail values are used to construct the score; the final test p-value comes from the outer permutation.

Our additional controlled copying and near-copy experiments also suggest that negative ZWZ_W responses may provide information about candidate-reference overlap. Whether these responses can be developed into a dedicated memorization diagnostic, analogous to the dispersion diagnostic, remains a question for future work. More generally, we plan to investigate additional component-based diagnostics while retaining broad detection coverage as the goal of the overall score. We will clarify this current goal and the developing role of component-level interpretation in the revised paper.

CW2. Evaluation at larger sample sizes

Thank you for this very good point. We repeated the main experiments in Figs. 3–5 with 10k samples per group to examine whether the conclusions change at a larger sample size. These experiments assess detection, signed dispersion diagnosis, and severity ranking, respectively.

For Figs. 3 and 5, we retained the original signal settings and additionally evaluated weaker signals to examine differences that could be obscured by saturation. For detection, we targeted weaker signals with the highest rejection rate among the three methods above 80% but not saturated. For severity ranking, we added finer weak-signal ladders. We report the original- and weak-signal results separately below.

Detection at 10k (Fig. 3). We extended Fig. 3 to 10k samples per group to examine whether the detection findings in the paper persist at a larger sample size. FID, KID, and standard six-arm ZID shared the inputs and 499 random relabelings. Both tables report 5%-level rejection rates, with 300 repetitions per condition.

With no perturbation, the rejection rates were 7.0% for FID, 4.7% for KID, and 6.7% for ZID.

Original signal settings.

ConditionFIDKIDZID
Location100.0%100.0%100.0%
Dispersion100.0%82.3%100.0%
Linear dependence100.0%100.0%100.0%
Nonlinear dependence5.3%100.0%100.0%
Multimodality4.7%100.0%100.0%
Off-manifold perturbation100.0%11.0%100.0%

At the original signal strengths, ZID reaches a 100% rejection rate in all six settings, while FID and KID each reach 100% in four. FID still does not respond to nonlinear dependence or multimodality even with 10k samples per group, as expected because these departures affect higher moments rather than the means and covariances captured by FID. KID catches up for dispersion at the original signal strength as the sample size increases to 10k, but its rejection rate for off-manifold perturbations remains low. We also examine weaker signal settings below.

Weaker signal settings. We used the highest rejection rate among the three methods in pilot runs as the signal-selection criterion, targeting above 80% but not saturated. The table reports rejection rates from 300 formal repetitions per condition.

ConditionFIDKIDZID
Location71.0%97.7%96.3%
Dispersion5.0%3.7%97.3%
Linear dependence99.3%68.3%98.0%
Nonlinear dependence6.0%10.0%92.7%
Multimodality7.3%7.7%100.0%
Off-manifold perturbation11.7%5.3%98.7%
AuthorsOpenReview ↗

Response to Bao4 (2/4): Dispersion and ranking at larger sample sizes

Signed dispersion diagnosis at 10k (Fig. 4). We extended Fig. 4 to 10k samples per group, and the key findings remain the same as in the paper: ZID distinguishes contraction from expansion through the sign of its dispersion score, whereas FID does not indicate the direction of the dispersion change.

The following table reports means over eight repetitions per scale, with 10k samples per group. The scale cc multiplies candidate deviations from the feature-pool mean: c<1c<1 contracts the candidate sample and c>1c>1 expands it. The ZID column is the ungated net signed dispersion score, not the overall ZID score or a significance-qualified direction call.

Scale (c)FIDZID net signed dispersion score
0.7218.49-6,529.51
0.8011.56-4,741.70
0.906.33-1,708.27
1.004.93+0.50
1.107.34+1,457.21
1.2013.57+3,793.59
1.3023.63+5,523.40

Severity ranking at 10k (Fig. 5). We extended Fig. 5 to 10k samples per group to examine whether the metrics still rank increasing perturbation severity correctly. At the original signal strengths, ZID matches or exceeds both baselines in mean severity-ranking correlation across all six departure families. The following tables report mean Spearman correlations over 100 repetitions, each with six ordered severities; this comparison uses metric scores rather than permutation rejections.

Original six severity levels. The following table reports the results at the original severity levels.

DepartureFIDKIDZID
Location1.0001.0001.000
Dispersion (collapse).991.9701.000
Linear dependence1.0001.0001.000
Nonlinear dependence-.0691.0001.000
Multimodality-.081.971.991
Off-manifold support.999.9471.000

Weaker signal settings. To avoid saturation, we also used weaker signals. The following table reports the mean Spearman correlations under these weaker settings.

DepartureFIDKIDZID
Location.890.945.919
Dispersion (collapse).355.225.987
Linear dependence.932.864.869
Nonlinear dependence-.030.822.893
Multimodality-.092-.010.830
Off-manifold support.633.002.891

ZID has the highest mean correlation for dispersion, nonlinear dependence, multimodality, and off-manifold support. KID has the highest mean for location, and FID for linear dependence. This pattern is the same as in the paper.

For better visualization, we plot the above new results on an anonymous figure page.

The controlled experiments use a fixed pool of 50k real CIFAR-10 image features. Preserving their disjoint-sampling design would require at least 100k observations to form two groups of 50k. To examine ranking stability at that larger scale, we therefore used Stein et al.’s (2023) released generator samples, comparing results at 500, 10k, and 50k samples per group.

Ranking stability at 500, 10k, and 50k.

We used 41 models from Stein et al.’s (2023) released datasets (13 CIFAR-10, 9 FFHQ, 11 ImageNet, and 8 LSUN Bedroom), with the same DINOv2 features for all three metrics. The samples were nested, with a common real reference across models within each dataset at each sample size. We counted pairwise ranking reversals relative to each metric’s 50k ranking. The table reports mean ± sample SD over 20 fixed-seed nested draws at 500; the 10k comparison used one fixed draw.

Dataset (500 vs 50k per group)Model pairsDINOv2 FDKIDZID
CIFAR-10782.15 ± 1.272.00 ± 0.971.60 ± 0.88
FFHQ360.30 ± 0.571.50 ± 0.890.35 ± 0.59
ImageNet557.25 ± 3.271.55 ± 0.761.75 ± 0.79
LSUN Bedroom280.70 ± 0.470.25 ± 0.440.70 ± 0.73

For 10k vs 50k, the only changes were one pair each for KID and ZID on CIFAR-10. For 500 vs 50k, KID and ZID rankings were stable across datasets, while DINOv2 FD showed more reversals on ImageNet but remained quite stable on the other three datasets.

We will include the 10k detection and severity-ranking results at original and weaker signal strengths, the 10k signed-dispersion results, and the 500/10k/50k ranking-stability comparison in the revised paper.

Dispersion without Precision/Recall

To check ZID's dispersion interpretation without relying on Precision or Recall, we compared its direction with that of covariance trace on the DiT samples. Covariance trace is the sum of the feature-wise variances and measures total feature dispersion. A larger trace than the real reference indicates larger dispersion, and a smaller trace indicates smaller dispersion. Both methods used the same 500 real and 500 generated images per class. The following table compares their directions relative to the real reference.

CFGZID dispersion directionCovariance-trace directionClasses agreeing
1LargerLarger6/6
2SmallerSmaller6/6
4SmallerSmaller6/6
8SmallerSmaller6/6
AuthorsOpenReview ↗

Response to Bao4 (3/4): Dispersion diagnosis—human evaluation and outlier sensitivity

Dispersion without Precision/Recall (continued).

The two methods agree in all six ImageNet classes at each of the four CFG settings used in the paper: larger dispersion at CFG 1 and smaller dispersion at CFG 2, 4, and 8 relative to real images. This agreement corroborates ZID's dispersion interpretation; it does not validate or replace Precision or Recall.

Independent human evaluation of dispersion direction. To check the dispersion interpretation without using Precision/Recall as the reference, we used Rapidata's public output-diversity matchups (Rapidata, 2026). In each matchup, raters see two grids, each with four images generated by a different model for the same prompt. Raters select the grid whose images look more similar to one another, indicating which grid they find less diverse. Of the 47,080 matchups in our collected data, 17 had tied human votes. We evaluated each method's direction against the human majority in all 47,063 remaining matchups.

We evaluated ZID’s directional diagnostic against human judgments of diversity. The comparison includes specialized measures: covariance trace for dispersion and Vendi for diversity. All methods use the same DINOv2 features. The first row uses the dispersion diagnostic defined in the paper. For the other three rows, we use a common decision rule: apply a two-sided permutation test to each method’s signed readout and report its direction only when the test rejects at the 5% level; otherwise, report no direction. ZID’s readout is the sum of its three standardized dispersion contrasts. For covariance trace and cosine Vendi, the readout is the difference between the two groups’ covariance traces or Vendi scores, respectively. In each case, the sign determines which group is assigned greater dispersion or diversity.

Directional methodAgreed with humansDisagreed with humansNo directionDirection assigned (%)Agreement when assigned
ZID (paper diagnostic)12,2512,25632,55630.82%84.45%
ZID dispersion component (common rule)12,5382,32832,19731.59%84.34%
Covariance trace (common rule)10,2611,57935,22325.16%86.66%
Cosine Vendi (common rule)7,39698638,68117.81%88.24%

With four images per group, there are (84)=70\binom{8}{4}=70 balanced relabelings. The two-sided p-values increase in steps of 2/702/70: 2.86%, 5.71%, and so on. Thus, a nominal 5% threshold only permits p-values up to 2.86%. Together with the small samples, this helps explain why most matchups received no direction at 5%. If we increase the significance level to 20%, in the order shown in the table, direction assignment rates are 63.73%, 65.67%, 54.11%, and 51.34%; agreement rates among assigned cases are 79.35%, 78.91%, 81.44%, and 81.87%.

These results provide a check of ZID's dispersion interpretation against human judgments of within-prompt diversity, rather than against Precision/Recall. Under the common rule, ZID assigns directions in more matchups than trace or Vendi, but with lower agreement among assigned cases, reflecting a coverage–agreement trade-off. We will include this human comparison in the revised paper.

As a supplementary check, we used two groups of 500 Inception-2048 observations, with independent outlier probabilities of 0%, 1%, or 5% in each group. Outliers had uniformly sampled directions at eight reference-pool RMS radii from the mean of the reference feature pool; outlier counts were not fixed or matched across groups.

When both groups were sampled independently from the same contaminated distribution, ZID's rejection rates at the 5% level were 5.28%, 5.18%, and 5.20%, respectively, over 5,000 repetitions per setting with 499 random relabelings.

We then checked whether the dispersion diagnostic retained the known direction when the generated group was contracted or expanded about the same reference mean, using scales of .8 and 1.2, respectively. ZID, covariance trace, and cosine Vendi used the same samples and 499 random relabelings, with 100 repetitions per setting. ZID uses the paper's standard diagnostic; trace and Vendi report a direction when their two-sided permutation tests reject at 5%. The table reports the percentage of all repetitions in which each method assigned the correct direction.

Outlier probabilityImposed change (scale)ZID correct directionTrace correct directionVendi correct direction
0%Contraction (0.8)100%100%100%
0%Expansion (1.2)100%100%100%
1%Contraction (0.8)100%41%100%
1%Expansion (1.2)100%35%100%
5%Contraction (0.8)100%57%100%
5%Expansion (1.2)100%36%100%

ZID and Vendi assigned the correct direction in every repetition; trace sometimes returned no direction under contamination but never assigned an incorrect direction. These results support ZID's directional response under this specified contamination model. We will include these supplementary outlier checks in the revised paper.

Reviewer_QTGnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ZID, a statistical diagnostic framework for generative model evaluation. FID depends on the first two feature moments and can miss higher-order distributional differences, while FID and KID are invariant to exchanging the two samples and do not themselves identify the direction of a dispersion change. ZID combines six standardized components from the rank-based RISE test and Gaussian-kernel GPK tests at two bandwidths. It reports a ranking score, a permutation-calibrated p-value for distributional equality, and a signed dispersion diagnosis when the component signs support an assignment. The authors evaluate ZID through controlled statistical and feature-space experiments, matched-moment stress tests, and pretrained image-generator evaluations. The results demonstrate broad sensitivity across the tested departures and additional information about under- and over-dispersion.

Critical strengths

1.This paper proposes a comprehensive statistical evaluation framework, which distinguishes three objectives: model ranking, testing distributional equality and dispersion direction diagnosis. Compared with the scalar discrepancies reported by FID and KID, ZID provides additional information through calibrated equality testing and a signed dispersion readout.

2.Systematic controlled experiments are carried out, covering various feature-space departure, such as location departure, scale change, dependence change, and multimodality changes, and compared with several existing indicators. These experiments better illustrate the detection ability of ZID in some scenarios where traditional indicators are prone to fail.

Critical weaknesses
  1. The scalability of ZID remains insufficiently characterized for large-sample evaluation and repeated checkpoint comparisons. Appendix B.1 usefully distinguishes score-only computation from the complete permutation test, but does not provide matched runtime and memory comparisons with FID and KID. Larger-sample measurements are needed to assess the practical cost of pairwise similarity construction and the additional cost of permutation calibration.

  2. The controlled experiments establish useful diagnostic capabilities, but the evidence that these capabilities improve practical model-selection decisions remains limited. The matched-moment optimization experiment is a valid stress test of FID, yet it does not establish how often such failures affect ordinary generator comparisons. Several pretrained-generator sweeps also produce similar rankings across ZID and conventional metrics; for example, the class-aggregated DiT results select CFG=2 under FID, KID, and ZID. The practical contribution would be clearer with cases showing how ZID’s additional information changes an evaluation decision and with independent evidence supporting that decision.

Questions for authors
  1. Please report matched runtime and peak-memory comparisons with FID and KID on the same hardware, separating ZID score-only computation from the complete permutation test. Please also evaluate scalability at larger sample sizes, such as 10,000 and 50,000 samples per set.

  2. For realistic generator comparisons where ZID and FID/KID disagree, please provide independent evidence supporting ZID’s ranking or dispersion diagnosis. What practical evaluation or model-selection decision changes as a result of this additional information?

  3. The pretrained-generator evaluations focus on images. Which parts of the framework extend directly to other generative domains, and what requirements would the feature representation and similarity construction need to satisfy?

3 comments
AuthorsOpenReview ↗

Response to QTGn (1/2): Computational cost and model selection

Thank you for your constructive comments. We report matched timing and memory measurements, then examine how metric choice affects model ranking and whether dispersion directions agree with human judgments.

Q1. Runtime and peak memory

We benchmarked ZID, FID, and KID on matched, precomputed Inception-2048 features with one CPU thread per run on a dual-socket AMD EPYC 9124 server (64 logical CPUs, 503 GiB RAM visible to Linux). Entries give wall time / peak RSS: time excludes feature loading and extraction; RSS includes loaded features. KID uses the full-sample unbiased estimator. Tests use the observed score and 499 random relabelings to compute a p-value. ZID uses statistic-preserving optimizations not in the submitted code (Anonymous ZID timing package: source, replay check, and timing instructions).

Samples per groupMethodScore onlyPermutation test
500FID2.67 s / 0.34 GiB22.41 min / 0.37 GiB
500KID0.05 s / 0.10 GiB0.12 s / 0.12 GiB
500ZID (full-matrix storage)0.12 s / 0.18 GiB0.17 s / 0.18 GiB
10kFID4.47 s / 0.62 GiB39.01 min / 1.24 GiB
10kKID23.55 s / 0.49 GiB40.58 s / 0.86 GiB
10kZID (full-matrix storage)31.33 s / 3.74 GiB41.39 s / 3.73 GiB
50kFID0.19 min / 2.44 GiB94.76 min / 5.55 GiB
50kKID9.72 min / 1.99 GiB16.82 min / 3.94 GiB
50kZID (low-memory version)22.48 min / 3.91 GiB25.80 min / 3.91 GiB

Use score-only computation for model ranking and permutation tests for calibrated equality decisions. At 50k, ZID (upper-triangle storage) takes 19.55 min / 40.65 GiB for the permutation test: faster than low-memory ZID, but more memory-intensive. FID scores faster at 10k and 50k. KID and ZID reuse label-independent pairwise work across relabelings; FID can reuse group moments, but its covariance square-root term changes. Our FID permutation implementation recomputes that term, so the reported times need not hold for a more optimized way for FID permutation test. We will add these timing and peak-memory comparisons and optimized implementation details to the revised paper.

Q2. Evidence for model ranking and dispersion diagnosis

Thank you for this very good point. We used two benchmarks with independent human judgments: Stein et al.’s (2023) ImageNet ratings to assess model rankings, and Rapidata’s output-diversity matchups to assess dispersion direction. We will add both evaluations to the revised paper.

Model ranking and selection

Stein et al.’s study used pools of 2,000 real images per dataset and 2,000 generated images per model. Each participant judged whether 100 real and 100 generated images sampled from these pools looked real. Models ranked higher when their generated images were more often mistaken for real ones, with rates averaged across participants.

For the 11 ImageNet models, image-level records cover only images participants actually rated: approximately 1.4–1.5k distinct generated images per model. The full 2,000-image real pool was available, but the identities of rated real images were not.

We compared the full real-image pool with 1.4k generated images per model, sampled without replacement to keep generated sample sizes equal across models. We repeated the sampling with 100 random seeds. We used DINOv2 ViT-L/14 features, following Stein et al.’s finding of better agreement with human judgments than with Inception features.

The table reports Spearman correlations between metric and human rankings (mean ± sample SD over 100 draws). Higher correlations indicate closer agreement.

DINOv2 FDKIDZID
.830 ± .018.856 ± .019.863 ± .018

ZID–FD ranking disagreements on ImageNet. ZID and FD disagreed on 3–5 of 55 pairs per draw. The table lists all five pairs observed across the 100 draws, ordered by absolute human gap. Each “choice” is the preferred model when the metrics disagreed. The gap is the judged-real percentage for ZID’s choice minus that for FD’s choice; positive values favor ZID. Participant-level bootstrap 95% CIs were adjusted for all 55 pairs.

ZID choiceFD choiceDisagreements / 100Human gap (pp)Adjusted 95% CI
ADMStyleGAN-XL100+11.72[3.66, 19.78]
ADMMask-GIT42+7.60[-2.15, 17.35]
ADMGLDM51-6.43[-15.03, 2.18]
ADMG-ADMULDM100-2.55[-12.79, 7.70]
GigaGANMask-GIT99+0.92[-8.28, 10.12]

Human point estimates favor ZID in three of these five pairs and FD in two. Only ADM vs. StyleGAN-XL has an adjusted interval that excludes zero: human judgments favor ADM, which both ZID and KID rank higher in all 100 draws, while FD ranks StyleGAN-XL higher in all 100.

Metric scores below are means over the same 100 draws; human ratings are unchanged across draws.

ReadoutADMStyleGAN-XL
Human judged-real fraction (higher better).2792.1620
DINOv2 FD (lower better)627.46617.34
KID (lower better).1845.2040
ZID (lower better)56,41671,655
AuthorsOpenReview ↗

Response to QTGn (2/2): Dispersion diagnosis and transfer beyond images

Direct ZID-versus-KID check. Across the same 55 ImageNet model pairs, ZID and KID disagreed on 0–2 pairs per draw: GigaGAN vs. Mask-GIT in 84 of 100 draws, and ADMG vs. LDM in two. In all 84 GigaGAN–Mask-GIT disagreements, ZID preferred GigaGAN and KID preferred Mask-GIT. Human ratings favor GigaGAN by 0.92 percentage points, but the adjusted 95% CI is [-8.28, 10.12]. For ADMG vs. LDM, each metric preferred ADMG in one of the two disagreements. Human ratings favor LDM by 6.43 points; the adjusted CI for ADMG minus LDM is [-15.03, 2.18]. Neither human preference is statistically significant after adjustment for all 55 pairs.

Dispersion diagnosis: does the direction agree with human judgments?

Rapidata’s public output-diversity matchups (Rapidata, 2026) show raters two grids of four images from different models responding to the same prompt. Raters choose the grid whose images look more similar to one another, indicating which grid they find less diverse. Of the 47,080 matchups, 17 had tied human votes. We evaluated directions against the human majority in all 47,063 remaining matchups.

We evaluated ZID’s directional diagnostic against human judgments of diversity. The comparison includes specialized measures: covariance trace for dispersion, Vendi for diversity, and mean within-group LPIPS for perceptual variation. ZID, covariance trace, and cosine Vendi (Friedman and Dieng, 2023) use the same DINOv2 features; mean LPIPS uses standard LPIPS (AlexNet, v0.1; Zhang et al., 2018). The first row uses the paper’s ZID diagnostic. The last four use a common decision rule: report a direction only if the readout’s two-sided permutation test rejects at 5%; otherwise, report no direction. The readouts are the sum of ZID’s three standardized dispersion contrasts, or the between-group difference in covariance trace, cosine Vendi, or mean within-group LPIPS. When the test rejects, the sign of the readout determines which group is assigned greater dispersion or diversity.

Directional methodAgreed with humansDisagreed with humansNo directionDirection assigned (%)Agreement when assigned
ZID (paper diagnostic)12,2512,25632,55630.82%84.45%
ZID dispersion component (common rule)12,5382,32832,19731.59%84.34%
Covariance trace (common rule)10,2611,57935,22325.16%86.66%
Cosine Vendi (common rule)7,39698638,68117.81%88.24%
Mean LPIPS (common rule)13,6923,23630,13535.97%80.88%

With four images per group, there are (84)=70\binom{8}{4}=70 balanced relabelings. The two-sided p-values increase in steps of 2/702/70: 2.86%, 5.71%, and so on. Thus, a nominal 5% threshold only permits p-values up to 2.86%. Together with the small samples, this helps explain why most matchups received no direction at 5%.

The following table applies the same rules at 10% and 20%. Each entry gives direction assigned (%) / agreement with the human majority when assigned (%).

Directional method10%20%
ZID (paper diagnostic)47.49% / 82.10%63.73% / 79.35%
ZID dispersion component (common rule)47.98% / 81.92%65.67% / 78.91%
Covariance trace (common rule)39.53% / 84.01%54.11% / 81.44%
Cosine Vendi (common rule)32.59% / 85.45%51.34% / 81.87%
Mean LPIPS (common rule)50.77% / 78.59%63.84% / 76.12%

Under the common rule, ZID assigned directions more often than trace or Vendi at each level, but had lower agreement when assigned, reflecting a coverage–agreement trade-off. Compared with LPIPS, ZID’s dispersion component had higher agreement when assigned at all three levels; at 20%, it also had a higher assignment rate.

Q3. Transfer beyond images

Yes. The rank-based and Gaussian-kernel components, their aggregation into the ZID score, and the signed dispersion readout are not specific to images. These parts can be used with other types of observations once suitable pairwise similarities or dissimilarities are available.

What changes across domains is how those pairwise relations are constructed. For text or audio, one could use distances between suitable embeddings; if a meaningful distance is already available on the original observations, no additional representation is required. The dispersion readout then describes variation under the chosen similarity or dissimilarity.

References

Friedman, D., and Dieng, A. B. (2023). The Vendi Score: A Diversity Evaluation Metric for Machine Learning. Transactions on Machine Learning Research.

Rapidata. (2026). Output Diversity (Mode Collapse) benchmark: public matchups. benchmark.ai.

Stein, G., et al. (2023). Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems, 36.

Zhang, R., et al. (2018). The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 586–595.

AuthorsOpenReview ↗

Follow-up on computational cost and practical evaluation value

Thank you again for your thoughtful review and helpful suggestions! In response to your scalability question, we report matched timing and peak-memory comparisons with FID and KID up to 50k samples per group, separating score-only computation from complete tests with 499 random relabelings. To address your question about practical model-selection value, we compare metric rankings with independent human realism judgments from Stein et al. (2023). This includes a concrete case where ZID and FD select different models, with human judgments significantly favoring ZID’s choice after adjustment for multiple comparisons. We also evaluate the dispersion diagnostic against 47,063 human-rated diversity matchups, alongside comparisons with specialized diversity measures, and clarify how the framework can extend beyond images. We would really appreciate hearing your thoughts on these results and clarifications, and would be happy to discuss any aspect further.

Reviewer_xJDPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In the paper, the authors propose ZID (ZZ-resolved Integrated Diagnostic), a method for evaluating generative models that combines six standardized statistics, built from rank-weighted RISE representation and Gaussian-kernel GPK at two bandwidths. It reports three things: a departure score for ranking, a permutation pp-value for equality test, and whether the generated samples are under-dispersed (mode collapse) or over-dispersed. The conducted experiments, which cover several controlled types of feature-space departure and pretrained generative model families, demonstrate the effectiveness of the proposed method.

Critical strengths
  • ZID provides a permutation p-value and signed dispersion readout that separates mode collapse (under-dispersion) from over-dispersion, which no symmetric scalar metric (like FID or KID) can do.

  • As shown in Fig. 3, ZID is the only method in the comparison that detects all controlled departure types reliably, with at least 0.70 power on each. Competing metrics excel on some types and fail on others.

Critical weaknesses
  • ZID is blind to memorization, so it may rate a model that simply copies its training data as perfect. Moreover, the experiments focus largely on settings designed to expose the weaknesses of selected existing metrics, rather than on a broad benchmark. An extensive comparison, like that of Stein et al. (NeurIPS 2023), is missing.

  • The computational cost of ZID is not reported, and its feasibility at sample sizes standard in evaluation of generative models (e.g., 10k-50k) is not demonstrated. The building blocks of ZID are known, so the novelty lies mainly in their aggregation and the directional readout, which may be considered incremental.

Questions for authors
  • What are the computational cost and memory requirements of ZID at sample sizes standard in evaluation of generative models (10k-50k)?

  • Could the framework be extended to detect memorization, for example through a train-versus-test comparison as in FLD (Jiralerspong et al., NeurIPS 2023)?

  • How was the 0.175 multiplier for the GPK-small bandwidth selected?

6 comments
AuthorsOpenReview ↗

Response to xJDP (1/3): Computational cost and training-data copying

Thank you for your constructive comments. We measured cost at 10k and 50k, tested how scores respond to training-image copies, and broadened the comparison using Stein et al.'s released data.

1. Computational cost

We measured both single scores and permutation tests at 500, 10k, and 50k samples per group. The ZID timings use optimized implementations that preserve the statistic and differ from the originally submitted code (Anonymous ZID timing package: source code, replay check, and timing instructions). We used one CPU thread per run on a dual-socket AMD EPYC 9124 server (64 logical CPUs, 503 GiB of RAM visible to Linux), with matched, precomputed Inception-2048 features. We report wall time and peak RSS. Wall time excludes feature loading and extraction; peak RSS includes the loaded features. KID uses the full-sample unbiased estimator. A permutation test includes the observed score, 499 random relabelings, and the resulting p-value.

Samples per groupMethodScore onlyPermutation test
500FID2.67 s / 0.34 GiB22.41 min / 0.37 GiB
500KID0.05 s / 0.10 GiB0.12 s / 0.12 GiB
500ZID (full-matrix storage)0.12 s / 0.18 GiB0.17 s / 0.18 GiB
10kFID4.47 s / 0.62 GiB39.01 min / 1.24 GiB
10kKID23.55 s / 0.49 GiB40.58 s / 0.86 GiB
10kZID (full-matrix storage)31.33 s / 3.74 GiB41.39 s / 3.73 GiB
50kFID0.19 min / 2.44 GiB94.76 min / 5.55 GiB
50kKID9.72 min / 1.99 GiB16.82 min / 3.94 GiB
50kZID (low-memory version)22.48 min / 3.91 GiB25.80 min / 3.91 GiB

For routine model ranking, use score-only computation; use the full permutation test when a calibrated equality decision is needed. At 50k, ZID (upper-triangle storage) runs the permutation test in 19.55 min with 40.65 GiB peak RSS, trading more memory for speed relative to the low-memory version. FID is faster for scoring at 10k and 50k. KID and ZID reuse label-independent pairwise computations; FID can reuse work in group moments, but its covariance square-root term changes with relabeling. Our FID Permutation Test implementation recomputes that term for each relabeling, so the reported times need not hold for a more optimized way for FID permutation test. We will add these timing and peak-memory comparisons, together with details of the optimized implementations, to the revised paper.

2. Training-data copying and memorization

What happens when generated samples copy the evaluation reference? In our controlled comparisons, reference copying did not receive a perfect ZID score. This addresses the reference-overlap case of the reviewer’s concern; it does not establish that ZID detects memorization of training examples absent from the evaluation reference. We use Inception-2048 features from 20 ImageNet classes, 500 observations per group, and five seeds. Entries are means ± sample SD (unbiased KID shown x1000).

Candidates copied from referenceFIDKID x1000ZID
0%70.18 ± .71-.072 ± .242.99 ± .95
10%63.63 ± .47-.237 ± .10813.45 ± 1.30
25%53.29 ± .30-.490 ± .04947.63 ± 1.24
50%36.25 ± .50-.845 ± .046107.56 ± 1.17
100%approx. 0-1.310 ± .031226.65 ± 2.43

Copying puts each copy and its original in different groups. These close pairs therefore contribute to cross-group rather than within-group similarity. Random relabeling would place some of those close pairs within a group, so the observed within-group similarity can be lower than expected. ZID treats this deficit, as well as an excess, as evidence of a departure, so the deficit raises the score.

As a robustness check, adding Gaussian noise to the copied features (SD: 1% of the training SD in each coordinate) preserved the pattern: ZID rose with the copy fraction, while FID and KID fell. We will include the controlled copying experiment and its noise check, and explain why cross-group close pairs increase ZID, in the revised paper.

Can ZID be used in a train-versus-test comparison? Yes. As with FID and KID, one can score the same outputs against equally sized training and held-out references in the same feature space and report the difference descriptively. An FLD-style detector (Jiralerspong et al., 2023) would require separate validation.

AuthorsOpenReview ↗

Response to xJDP (2/3): Benchmark breadth and ranking stability

3. Benchmark breadth

Thank you for pointing us to Stein et al. (2023). Their human realism study used pools of 2,000 real images per dataset and 2,000 generated images per model. Each participant judged whether 100 real and 100 generated images sampled from these pools looked real. Models ranked higher when their generated images were more often mistaken for real ones, with rates averaged across participants.

We could not verify the released real-image pool used for CIFAR-10, and one LSUN model lacked per-image ratings. We therefore included 27 models: 9 FFHQ, 11 ImageNet, and 7 LSUN Bedroom. Image-level records cover only images participants actually rated: approximately 1.4–1.5k distinct generated images per model. The full 2,000-image real pools were available, but the identities of rated real images were not.

We compared the full real-image pools with 1.4k generated images per model, sampled without replacement to keep generated sample sizes equal across models. We repeated the sampling with 100 random seeds. We computed FD, KID, and ZID on DINOv2 ViT-L/14 features, following Stein et al.’s finding of better agreement with human judgments than with Inception features.

Agreement with human realism judgments. The table reports the mean ± sample SD of the Spearman correlation between metric and human rankings over the 100 draws for each dataset. Higher correlations indicate closer agreement.

DatasetNumber of modelsDINOv2 FDKIDZID
FFHQ9.300 ± .000.211 ± .031.234 ± .048
ImageNet11.830 ± .018.856 ± .019.863 ± .018
LSUN Bedroom71.000 ± .0001.000 ± .0001.000 ± .000

Why is agreement low on FFHQ? The low Spearman correlations on FFHQ prompted us to look more closely at its model rankings. We found that some models rated highly by humans were substantially under-dispersed. For example, StyleGAN2-ADA ranked third in human realism but had the third-lowest total feature variance, which is the sum of the variances across all DINOv2 feature dimensions, among the nine FFHQ models—only 79.58% of the real-data value. InsGen ranked second in human realism despite retaining only 81.52%. FID, KID, and ZID all ranked these two models substantially lower than the human evaluation did. By contrast, the three highest-rated ImageNet models had total feature variances between 97.57% and 100.03% of the real-data value. Across all ImageNet models except the lowest-rated model, the range was 92.17%–100.03%; the lowest-rated model, BigGAN, retained 89.03%.

ZID–FD ranking disagreements on ImageNet. ZID and DINOv2 FD disagreed on 3–5 of 55 model pairs per draw. The table includes all five pairs with disagreements across the 100 draws, ordered by absolute human gap. Each metric’s “choice” is its preferred model when they disagreed. The human gap is the difference in judged-real percentages (ZID choice minus FD choice); positive values favor ZID’s choice. Participant-level bootstrap 95% CIs were adjusted for all 55 pairs.

ZID choiceFD choiceDisagreements / 100Human gap (pp)Adjusted 95% CI
ADMStyleGAN-XL100+11.72[3.66, 19.78]
ADMMask-GIT42+7.60[-2.15, 17.35]
ADMGLDM51-6.43[-15.03, 2.18]
ADMG-ADMULDM100-2.55[-12.79, 7.70]
GigaGANMask-GIT99+0.92[-8.28, 10.12]

Human point estimates favor ZID in three of these five pairs and FD in two. Only ADM vs. StyleGAN-XL has an adjusted interval that excludes zero: human judgments favor ADM, which both ZID and KID rank higher in all 100 draws, while FD ranks StyleGAN-XL higher in all 100.

Sample-size stability. Using Stein et al.’s released datasets, we examined 41 models (13 CIFAR-10, 9 FFHQ, 11 ImageNet, 8 LSUN Bedroom) with nested samples of 500, 10k, and 50k images per group. We counted pairwise ranking reversals relative to each metric’s 50k ranking. The table reports mean ± sample SD over 20 fixed-seed nested draws at 500; the 10k comparison used one fixed draw.

Dataset (500 vs 50k per group)Model pairsDINOv2 FDKIDZID
CIFAR-10782.15 ± 1.272.00 ± 0.971.60 ± 0.88
FFHQ360.30 ± 0.571.50 ± 0.890.35 ± 0.59
ImageNet557.25 ± 3.271.55 ± 0.761.75 ± 0.79
LSUN Bedroom280.70 ± 0.470.25 ± 0.440.70 ± 0.73

For 10k vs 50k, the only changes were one pair each for KID and ZID on CIFAR-10. For 500 vs 50k, KID and ZID rankings were stable across datasets, while DINOv2 FD showed more reversals on ImageNet but remained quite stable on the other three datasets.

AuthorsOpenReview ↗

Response to xJDP (3/3): Dispersion diagnosis, bandwidth choice, and contribution

Human evaluation of dispersion direction. To assess diversity separately, we used Rapidata’s public output-diversity matchups (Rapidata, 2026). In each matchup, raters see two grids, each with four images generated by a different model for the same prompt. Raters select the grid whose images look more similar to one another, indicating which grid they find less diverse. Our snapshot contained 47,080 matchups. After excluding 17 with tied human votes, we evaluated each method’s direction against the human majority in the remaining 47,063 matchups.

We evaluated ZID’s directional diagnostic against human judgments of diversity. The comparison includes specialized measures: covariance trace for dispersion and Vendi for diversity. For these 47,063 matchups, all methods use the same DINOv2 features. The first row uses the dispersion diagnostic defined in the paper. For the other three rows, we use a common decision rule: apply a two-sided permutation test to each method’s signed readout and report its direction only when the test rejects at the 5% level; otherwise, report no direction. ZID’s readout is the sum of its three standardized dispersion contrasts. For covariance trace and cosine Vendi, the readout is the difference between the two groups’ covariance traces or Vendi scores, respectively. In each case, the sign determines which group is assigned greater dispersion or diversity.

Directional methodAgreed with humansDisagreed with humansNo directionDirection assigned (%)Agreement when assigned
ZID (paper diagnostic)12,2512,25632,55630.82%84.45%
ZID dispersion component (common rule)12,5382,32832,19731.59%84.34%
Covariance trace (common rule)10,2611,57935,22325.16%86.66%
Cosine Vendi (common rule)7,39698638,68117.81%88.24%

With four images per group, there are (84)=70\binom{8}{4}=70 balanced relabelings. The two-sided p-values increase in steps of 2/702/70: 2.86%, 5.71%, and so on. Thus, a nominal 5% threshold only permits p-values up to 2.86%. Together with the small samples, this helps explain why most matchups received no direction at 5%. If we increase the significance level to 20%, in the order shown in the table, direction assignment rates are 63.73%, 65.67%, 54.11%, and 51.34%; agreement rates among assigned cases are 79.35%, 78.91%, 81.44%, and 81.87%. Under the common rule, ZID assigned directions more often than trace or Vendi, but had lower agreement with the human majority among assigned cases, reflecting a coverage–agreement trade-off. A directional diagnostic can indicate which set of images appears more diverse; FID and KID do not provide that direction.

We will include the Stein human-ranking comparisons, the sample-size stability results, and the Rapidata validation of dispersion direction in the revised paper.

4. Bandwidth choice

Why 0.175? We chose a relatively small multiplier so that the finer Gaussian kernel would respond to more local differences than the median-bandwidth kernel. In a few pilot experiments, we found that 0.175 worked well. We kept it fixed for the reported comparisons and later checked sensitivity to nearby values. The results in Appendix F (Table 19) support a range of nearby multipliers rather than singling out 0.175 as optimal. Across seven multipliers from .10 to .25, minimum power across the six departures is .70–.71 and mean null rejection is .032–.047. In the revision, we will clarify that 0.175 was chosen in pilot experiments, while Appendix F, Table 19 evaluates sensitivity to nearby bandwidth values.

5. Contribution

What is new? ZID provides a single evaluation procedure that detects a broad range of distributional departures with a fixed score and also offers a signed diagnostic of dispersion differences. The contribution lies in this combined evaluation capability, rather than in new graph or kernel constructions or a new aggregation rule. In Fig. 3, the same fixed ZID score detects six kinds of departure without switching metrics or tuning the score to each alternative. It also retains a diagnostic that the overall score alone cannot provide: in Fig. 4, contraction and expansion can yield nearly the same FID, whereas ZID distinguishes their directions. On Rapidata’s benchmark, ZID’s dispersion diagnostic agreed with the human majority in most matchups where it assigned a direction.

The human-ranking comparison also illustrates how metric choice can affect model selection: ZID ranks ADM above StyleGAN-XL in all 100 draws, whereas FD consistently gives the opposite ranking. Human ratings favor ADM, with an adjusted 95% confidence interval entirely above zero.

References

Rapidata. (2026). Output Diversity (Mode Collapse) benchmark: public matchups. benchmark.ai.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.