- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper points out two problems of FID, which emerge when comparing only the first and second moments of real and fake feature distributions: (1) the inability to accurately distinguish between real and generated distributions, and (2) the failure to provide an analysis of aspects such as dispersion due to the reliance on a single evaluation value. To address these problems, the paper proposes a new metric called ZID (Z-resolved Integrated Diagnostic).
Critical strengths
The problems with FID highlighted in this paper are indeed valid. A compelling example is shown in Fig. 1, where unrealistic noise images yield a better FID score against real images than the FID score between two distinct sets of real images. The proposed evaluation method (ZID) serves as a convincing solution to the two identified problems, (1) the inability to accurately distinguish between real and generated distributions and (2) the failure to analyze aspects such as dispersion, despite the study's inherent limitations regarding low-dimensional embeddings and small sample sizes.
Critical weaknesses
CW1. The ZID score defined in this paper does not provide a consistent interpretation of differences between distributions; the meaning of the difference can vary with the experimental setup, creating ambiguity or difficulty for users attempting to interpret the score. For example, the ZID score takes the minimum over values 6r_(i)/i (L215-219), so the final score is determined by a single arm’s value. Among the six r_k values, three are based on Z_W and the other three on Z_D. Therefore, depending on whether the final score is determined by Z_W or Z_D, the score may have different interpretations, making the meaning of the score difficult to interpret consistently.
CW2. It appears to require further validation regarding experimental fairness. First, the FID and KID scores are typically calculated using sample sizes of 10k to 50k, which are the metrics that ZID score is primarily compared. However, the sample sizes used for comparison are substantially smaller (e.g., n = 500) than those in the originally proposed setting, which could significantly affect the reported results. Second, regarding the use of precision and recall to measure fidelity and diversity (in Section 5), these metrics are known to be inherently vulnerable to outliers [1], so they may not provide reliable indicators in regimes such as high CFG, where the number of off-manifold samples increases. Thus, it remains questionable whether the paper’s claim would hold consistent when these fairness problems are resolved.
- [1] Naeem, Muhammad Ferjad, et al. "Reliable fidelity and diversity metrics for generative models." International conference on machine learning. PMLR, 2020.
Questions for authors
Q1. Does the ranking of existing generative models based on the FID score change when using the ZID score, and how can such changes in ranking be interpreted in the context of the method proposed in this paper?
Q2. Could the authors add statements explaining the design rationale in the sections introducing the proposed method, such as Section 3.3? Currently, the reader has to infer the author’s design intentions, which makes it difficult to assess the validity of the proposed method.
9 comments
Response to Bao4 (2/4): Dispersion and ranking at larger sample sizes
Signed dispersion diagnosis at 10k (Fig. 4). We extended Fig. 4 to 10k samples per group, and the key findings remain the same as in the paper: ZID distinguishes contraction from expansion through the sign of its dispersion score, whereas FID does not indicate the direction of the dispersion change.
The following table reports means over eight repetitions per scale, with 10k samples per group. The scale multiplies candidate deviations from the feature-pool mean: contracts the candidate sample and expands it. The ZID column is the ungated net signed dispersion score, not the overall ZID score or a significance-qualified direction call.
| Scale (c) | FID | ZID net signed dispersion score |
|---|---|---|
| 0.72 | 18.49 | -6,529.51 |
| 0.80 | 11.56 | -4,741.70 |
| 0.90 | 6.33 | -1,708.27 |
| 1.00 | 4.93 | +0.50 |
| 1.10 | 7.34 | +1,457.21 |
| 1.20 | 13.57 | +3,793.59 |
| 1.30 | 23.63 | +5,523.40 |
Severity ranking at 10k (Fig. 5). We extended Fig. 5 to 10k samples per group to examine whether the metrics still rank increasing perturbation severity correctly. At the original signal strengths, ZID matches or exceeds both baselines in mean severity-ranking correlation across all six departure families. The following tables report mean Spearman correlations over 100 repetitions, each with six ordered severities; this comparison uses metric scores rather than permutation rejections.
Original six severity levels. The following table reports the results at the original severity levels.
| Departure | FID | KID | ZID |
|---|---|---|---|
| Location | 1.000 | 1.000 | 1.000 |
| Dispersion (collapse) | .991 | .970 | 1.000 |
| Linear dependence | 1.000 | 1.000 | 1.000 |
| Nonlinear dependence | -.069 | 1.000 | 1.000 |
| Multimodality | -.081 | .971 | .991 |
| Off-manifold support | .999 | .947 | 1.000 |
Weaker signal settings. To avoid saturation, we also used weaker signals. The following table reports the mean Spearman correlations under these weaker settings.
| Departure | FID | KID | ZID |
|---|---|---|---|
| Location | .890 | .945 | .919 |
| Dispersion (collapse) | .355 | .225 | .987 |
| Linear dependence | .932 | .864 | .869 |
| Nonlinear dependence | -.030 | .822 | .893 |
| Multimodality | -.092 | -.010 | .830 |
| Off-manifold support | .633 | .002 | .891 |
ZID has the highest mean correlation for dispersion, nonlinear dependence, multimodality, and off-manifold support. KID has the highest mean for location, and FID for linear dependence. This pattern is the same as in the paper.
For better visualization, we plot the above new results on an anonymous figure page.
The controlled experiments use a fixed pool of 50k real CIFAR-10 image features. Preserving their disjoint-sampling design would require at least 100k observations to form two groups of 50k. To examine ranking stability at that larger scale, we therefore used Stein et al.’s (2023) released generator samples, comparing results at 500, 10k, and 50k samples per group.
Ranking stability at 500, 10k, and 50k.
We used 41 models from Stein et al.’s (2023) released datasets (13 CIFAR-10, 9 FFHQ, 11 ImageNet, and 8 LSUN Bedroom), with the same DINOv2 features for all three metrics. The samples were nested, with a common real reference across models within each dataset at each sample size. We counted pairwise ranking reversals relative to each metric’s 50k ranking. The table reports mean ± sample SD over 20 fixed-seed nested draws at 500; the 10k comparison used one fixed draw.
| Dataset (500 vs 50k per group) | Model pairs | DINOv2 FD | KID | ZID |
|---|---|---|---|---|
| CIFAR-10 | 78 | 2.15 ± 1.27 | 2.00 ± 0.97 | 1.60 ± 0.88 |
| FFHQ | 36 | 0.30 ± 0.57 | 1.50 ± 0.89 | 0.35 ± 0.59 |
| ImageNet | 55 | 7.25 ± 3.27 | 1.55 ± 0.76 | 1.75 ± 0.79 |
| LSUN Bedroom | 28 | 0.70 ± 0.47 | 0.25 ± 0.44 | 0.70 ± 0.73 |
For 10k vs 50k, the only changes were one pair each for KID and ZID on CIFAR-10. For 500 vs 50k, KID and ZID rankings were stable across datasets, while DINOv2 FD showed more reversals on ImageNet but remained quite stable on the other three datasets.
We will include the 10k detection and severity-ranking results at original and weaker signal strengths, the 10k signed-dispersion results, and the 500/10k/50k ranking-stability comparison in the revised paper.
Dispersion without Precision/Recall
To check ZID's dispersion interpretation without relying on Precision or Recall, we compared its direction with that of covariance trace on the DiT samples. Covariance trace is the sum of the feature-wise variances and measures total feature dispersion. A larger trace than the real reference indicates larger dispersion, and a smaller trace indicates smaller dispersion. Both methods used the same 500 real and 500 generated images per class. The following table compares their directions relative to the real reference.
| CFG | ZID dispersion direction | Covariance-trace direction | Classes agreeing |
|---|---|---|---|
| 1 | Larger | Larger | 6/6 |
| 2 | Smaller | Smaller | 6/6 |
| 4 | Smaller | Smaller | 6/6 |
| 8 | Smaller | Smaller | 6/6 |
Response to Bao4 (3/4): Dispersion diagnosis—human evaluation and outlier sensitivity
Dispersion without Precision/Recall (continued).
The two methods agree in all six ImageNet classes at each of the four CFG settings used in the paper: larger dispersion at CFG 1 and smaller dispersion at CFG 2, 4, and 8 relative to real images. This agreement corroborates ZID's dispersion interpretation; it does not validate or replace Precision or Recall.
Independent human evaluation of dispersion direction. To check the dispersion interpretation without using Precision/Recall as the reference, we used Rapidata's public output-diversity matchups (Rapidata, 2026). In each matchup, raters see two grids, each with four images generated by a different model for the same prompt. Raters select the grid whose images look more similar to one another, indicating which grid they find less diverse. Of the 47,080 matchups in our collected data, 17 had tied human votes. We evaluated each method's direction against the human majority in all 47,063 remaining matchups.
We evaluated ZID’s directional diagnostic against human judgments of diversity. The comparison includes specialized measures: covariance trace for dispersion and Vendi for diversity. All methods use the same DINOv2 features. The first row uses the dispersion diagnostic defined in the paper. For the other three rows, we use a common decision rule: apply a two-sided permutation test to each method’s signed readout and report its direction only when the test rejects at the 5% level; otherwise, report no direction. ZID’s readout is the sum of its three standardized dispersion contrasts. For covariance trace and cosine Vendi, the readout is the difference between the two groups’ covariance traces or Vendi scores, respectively. In each case, the sign determines which group is assigned greater dispersion or diversity.
| Directional method | Agreed with humans | Disagreed with humans | No direction | Direction assigned (%) | Agreement when assigned |
|---|---|---|---|---|---|
| ZID (paper diagnostic) | 12,251 | 2,256 | 32,556 | 30.82% | 84.45% |
| ZID dispersion component (common rule) | 12,538 | 2,328 | 32,197 | 31.59% | 84.34% |
| Covariance trace (common rule) | 10,261 | 1,579 | 35,223 | 25.16% | 86.66% |
| Cosine Vendi (common rule) | 7,396 | 986 | 38,681 | 17.81% | 88.24% |
With four images per group, there are balanced relabelings. The two-sided p-values increase in steps of : 2.86%, 5.71%, and so on. Thus, a nominal 5% threshold only permits p-values up to 2.86%. Together with the small samples, this helps explain why most matchups received no direction at 5%. If we increase the significance level to 20%, in the order shown in the table, direction assignment rates are 63.73%, 65.67%, 54.11%, and 51.34%; agreement rates among assigned cases are 79.35%, 78.91%, 81.44%, and 81.87%.
These results provide a check of ZID's dispersion interpretation against human judgments of within-prompt diversity, rather than against Precision/Recall. Under the common rule, ZID assigns directions in more matchups than trace or Vendi, but with lower agreement among assigned cases, reflecting a coverage–agreement trade-off. We will include this human comparison in the revised paper.
As a supplementary check, we used two groups of 500 Inception-2048 observations, with independent outlier probabilities of 0%, 1%, or 5% in each group. Outliers had uniformly sampled directions at eight reference-pool RMS radii from the mean of the reference feature pool; outlier counts were not fixed or matched across groups.
When both groups were sampled independently from the same contaminated distribution, ZID's rejection rates at the 5% level were 5.28%, 5.18%, and 5.20%, respectively, over 5,000 repetitions per setting with 499 random relabelings.
We then checked whether the dispersion diagnostic retained the known direction when the generated group was contracted or expanded about the same reference mean, using scales of .8 and 1.2, respectively. ZID, covariance trace, and cosine Vendi used the same samples and 499 random relabelings, with 100 repetitions per setting. ZID uses the paper's standard diagnostic; trace and Vendi report a direction when their two-sided permutation tests reject at 5%. The table reports the percentage of all repetitions in which each method assigned the correct direction.
| Outlier probability | Imposed change (scale) | ZID correct direction | Trace correct direction | Vendi correct direction |
|---|---|---|---|---|
| 0% | Contraction (0.8) | 100% | 100% | 100% |
| 0% | Expansion (1.2) | 100% | 100% | 100% |
| 1% | Contraction (0.8) | 100% | 41% | 100% |
| 1% | Expansion (1.2) | 100% | 35% | 100% |
| 5% | Contraction (0.8) | 100% | 57% | 100% |
| 5% | Expansion (1.2) | 100% | 36% | 100% |
ZID and Vendi assigned the correct direction in every repetition; trace sometimes returned no direction under contamination but never assigned an incorrect direction. These results support ZID's directional response under this specified contamination model. We will include these supplementary outlier checks in the revised paper.
Response to Bao4 (1/4): Score interpretation and detection at 10k
Thank you for your constructive comments. We clarify the interpretation and design rationale of ZID, compare results at 500, 10k, and 50k, examine how metric choice affects model selection, and validate dispersion findings without relying on Precision/Recall.
CW1. Interpreting ZID when different components dominate
Thank you for this very good point. In practice, distributions can differ in many ways, and we usually do not know the type of change in advance. The goal of ZID is to cover as many of these changes as possible. This broad coverage is our main focus at the current stage and motivates an omnibus approach.
For interpretation, the values of the six arms may provide useful hints. Each arm is standardized relative to its permutation-null variation and mapped to the same two-sided normal-reference tail scale before combination. Thus, all comparisons use the same scoring rule, although different arms can explain the response in different settings. The dispersion diagnostic is one of our attempts in this direction, supported by the controlled experiments in the paper and the human-evaluation results discussed below under CW2. Developing further interpretations of the component values is an important next step.
Regarding the Simes combination, it can respond to either one particularly strong component or several moderately strong components. For example, reference-tail values (.001,1,1,1,1,1) give qrank=.006, with the minimum attained at i=1. Values (.003,.003,.003,1,1,1) give the same value, now with the minimum attained at i=3. Thus, the minimum need not represent an isolated single-arm response. These reference-tail values are used to construct the score; the final test p-value comes from the outer permutation.
Our additional controlled copying and near-copy experiments also suggest that negative ZW responses may provide information about candidate-reference overlap. Whether these responses can be developed into a dedicated memorization diagnostic, analogous to the dispersion diagnostic, remains a question for future work. More generally, we plan to investigate additional component-based diagnostics while retaining broad detection coverage as the goal of the overall score. We will clarify this current goal and the developing role of component-level interpretation in the revised paper.
CW2. Evaluation at larger sample sizes
Thank you for this very good point. We repeated the main experiments in Figs. 3–5 with 10k samples per group to examine whether the conclusions change at a larger sample size. These experiments assess detection, signed dispersion diagnosis, and severity ranking, respectively.
For Figs. 3 and 5, we retained the original signal settings and additionally evaluated weaker signals to examine differences that could be obscured by saturation. For detection, we targeted weaker signals with the highest rejection rate among the three methods above 80% but not saturated. For severity ranking, we added finer weak-signal ladders. We report the original- and weak-signal results separately below.
Detection at 10k (Fig. 3). We extended Fig. 3 to 10k samples per group to examine whether the detection findings in the paper persist at a larger sample size. FID, KID, and standard six-arm ZID shared the inputs and 499 random relabelings. Both tables report 5%-level rejection rates, with 300 repetitions per condition.
With no perturbation, the rejection rates were 7.0% for FID, 4.7% for KID, and 6.7% for ZID.
Original signal settings.
At the original signal strengths, ZID reaches a 100% rejection rate in all six settings, while FID and KID each reach 100% in four. FID still does not respond to nonlinear dependence or multimodality even with 10k samples per group, as expected because these departures affect higher moments rather than the means and covariances captured by FID. KID catches up for dispersion at the original signal strength as the sample size increases to 10k, but its rejection rate for off-manifold perturbations remains low. We also examine weaker signal settings below.
Weaker signal settings. We used the highest rejection rate among the three methods in pilot runs as the signal-selection criterion, targeting above 80% but not saturated. The table reports rejection rates from 300 formal repetitions per condition.