acceptodds
Under review as a conference paper at ICLR 2027

QID: Quantile Inception Distance for Generative Model Evaluation

Abstract

Generative model development requires repeated comparisons with a dataset of real images, often using only a small batch of generated samples at each checkpoint. We introduce Quantile Inception Distance (QID), a fast, sample-efficient metric that represents feature distributions by projected quantile displacements from a common reference. This optimal transport embedding provides coordinates for comparing generated distributions with the real dataset and with one another. We derive a finite-dimensional representation of this embedding, enabling efficient comparison of small generated batches with the full real dataset. We also derive fQID, a Gaussian approximation for faster scoring. We evaluate QID on over one million real and generated images across ImageNet, COCO and CIFAR-10. In small-batch ImageNet comparisons with Inception features, QID reduces pairwise ranking errors nearly fivefold relative to FID and roughly halves the variance of score differences between models relative to MIND. On ImageNet, QID achieves higher pairwise ranking agreement than polynomial and RBF kernel estimators with 2,000–10,000 generated images in both Inception and DINOv2 space. Speedups reach over full-reference kernel estimation and over FID.

Primary area:
generative models
Keywords:
generative model evaluation, optimal transport, sliced Wasserstein distance, quantile embeddings, distribution metrics, sample-efficient evaluation, fixed-reference evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TUr7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The Quantile Inception Distance (QID) method introduces a new evaluation metric for generative models. It is based on the concept of optimal transport embedding (using the Sliced Wasserstein distance, W2W_{2}) to project and compare feature distributions against a common real-world reference dataset. This method aims to compare generated images with original ones in particular in the low generated sample framework. Its main advantage lies in its ability to take into account higher order moments (not only mean and covariance like the Fréchet Inception Distance (FID)). This allows to detect subtle changes in the shape of distributions. Experiments have been done on Imagenet, COCO and CIFAR-10 databases. The results show improve in the pairwise ranking error of the chosen model.

Critical strengths

1- the evaluation is performed in the generated "low sample size" framework and proves interest. 2- The offline computation of the feature map on the real data set enables to have an efficient online algorithm for each new generated sample. 3- The Gaussian approximation increases this computational efficience without a significant deterioration of the results. 4- The theoretical foundations of this methods make it interesting.

Critical weaknesses

1- Although the method seems to perform better than FID, Figure 3 shows that on Imagenet, Polynomial MMD is more performant in partiuclar for the selection error. Can you explain this behaviour? 2- the authors mention the "low generated sample size" framework but are still working with several hundreds of generated samples. What would happen in a lower regime?

Questions for authors

In addition to the questions related to weaknesses, I would really appreciate if the authors can explain what would happen in the opposite regime: very low sample size as reference and large (or not necessarily) generated datasets.

3 comments
AuthorsOpenReview ↗

Response to TUr7 (part 1)

Thank you for your questions and and encouraging review. We are happy to continue the conversation below if the reviewer has any further or follow-up questions.

Why does polynomial MMD select the winner more reliably? Polynomial MMD's score gap between the leading models is larger relative to sampling variation (Figure 3). Reversing that pair can choose the wrong winner while leaving 54 of 55 pairs correct. At n=512 in Inception, this pair reverses in 157 of QID's 163 selection errors. We added a score-gap analysis of the common winner and runner-up, StyleGAN-XL and DiT-XL/2 with guidance 1.5. Their mean gap is 1.43 sampling standard deviations for polynomial MMD and 0.49 for QID. They recover this pair's ordering in 94.7% and 69.3% of trials, with selection errors of 5.5% and 31.8%. DINOv2 also gives polynomial MMD stronger leading-pair separation (Appendix G.4). At 3,000 directions, QID's primary Inception agreement rises to 95.5%, but selection error remains 29.1% (Appendix R.4).

Feature choice changes the realism of the models these metrics favor. In all six ImageNet bank–split settings, the common Inception winner has the panel's lowest observed human error (15.34%); the DINOv2 winner has the highest (32.96%). Higher error means real and generated images are harder to tell apart. We also evaluate 512-image selections against human judgments (Appendix T), averaging banks and splits. QID, polynomial MMD and RBF MMD selections have human errors of 20.42%, 16.24% and 21.21% in Inception, all below uniform choice (23.98%), and 32.95%, 32.96% and 32.96% in DINOv2.

What happens below hundreds of generated images? QID can still use the full real reference when few generated images are available. It exceeds MIND and Fréchet distance at n=8,16,32 in every bank and split; both kernels have higher agreement. At n=32, QID recovers 72.1% of pairs in Inception and 85.3% in DINOv2, exceeding MIND by 6.57 and 4.74 percentage points. To test whether the gains persist as fewer observations determine the generated quantiles, we added experiments at n=1,8,16,32,64 on all eleven ImageNet configurations, with 512 paired trials, both encoders, three banks and two splits (Appendix P.1).

With DINOv2 features, QID's small-batch rankings agree more closely with human judgments than MIND's. We use Stein et al.'s (2023) human judgments and report mean Spearman correlations for the primary bank and split below (Appendices O–P).

Method8 images16 images32 images
QID0.3800.5830.723
fQID0.4500.6200.740
MIND0.2230.4340.635
Fréchet distance−0.461−0.461−0.296
Polynomial MMD0.6230.7140.781
RBF MMD0.6370.7360.790

The 32-image QID–MIND gain remains positive after omitting any configuration in every bank and split. With all configurations, the primary comparison gives 0.088 (95% CI 0.064–0.099). Both kernels remain stronger, and Inception correlations are negative. The mean DINOv2 gain is 0.042 (0.034–0.044) in our expanded comparison of all 41 configurations across four datasets, averaging banks and splits (Table 2, Appendix O.3). With CLIP at 32 images and 100 directions, repeating the comparison gives 0.035 (95% CI 0.030–0.039; Appendix U). On ImageNet with DINOv2 features at 32 images, more directions bring QID close to kernel human agreement at lower scoring cost (Table 43).

The table above shows that fQID ranks these batches well using moments, while QID also detects the change preserving the feature mean and covariance (Table 4). At n=1, the generated quantile is constant and its variance is zero, so QID equals fQID. At this size, ranking recovery is 52.7% in Inception and 47.5% in DINOv2 for the primary bank and split (Appendix P.1, Figure 16). Unbiased covariance and kernel estimates require at least two observations and are omitted.

AuthorsOpenReview ↗

Response to TUr7 (part 2)

What if the real reference is small? More generated images refine the generated quantiles while leaving the real-reference quantiles unchanged. At N=8,n=5000, Inception QID recovers 94.8% of the small-reference ordering but only 59.4% of the full-reference ordering. At N=512,n=5000, full-reference agreement reaches 96.5% in Inception and 99.4% in DINOv2. Both kernels recover more full-reference pairs throughout N≤128. To separate the effects of real and generated sample sizes, we added experiments varying N=8,32,128,512 and n=32,1000,5000 and comparing recovery of the small-reference and full-reference orderings. We cross 32 reference draws with 32 generated trials (Appendix P.2). Intervals resample the reference and generated-trial axes separately.

The value of additional real data for selected-model realism depends on the feature representation. With DINOv2 and 5,000 generated images, enlarging the reference from N=8 to the full set raises QID's selected-model human error from 29.77% to 32.96%: +3.19 points (95% CI 1.07–5.61). Inception correlations remain negative; enlarging its reference lowers selected-model human error from 22.97% to 15.75%. We evaluate this reference-size grid against human judgments (Appendix T). At N=512,n=5000, QID and both kernels reproduce their full-reference DINOv2 choices in every observed trial. DINOv2 kernels remain stronger at N=8.

QID's representation holds for every positive N and n. With n>N, storing n coefficients per direction saves no space over N sorted reference values. When real data are plentiful, the averages and residual preserve the full-reference comparison in space proportional to the generated batch.

Dear reviewer TUr7,

With the discussion window ending today, we wanted to check in on your questions about very small samples. The revision adds experiments down to a single generated image (Appendix P.1), real references as small as 8 images for the opposite regime you asked about (Appendix P.2), and an analysis of why polynomial MMD selects the winner more reliably (Appendix G.4).

We hope these answer your questions and would welcome any final thoughts. We appreciate the care you took with the paper.

Kind regards, authors

Reviewer_25HUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Quantile Inception Distance (QID), an efficient metric for generative model evaluation based on sliced Wasserstein distance. Its main contribution is a compact representation of real-data distributions using averaged quantiles and residual terms, preserving the full-reference distance while reducing storage and computational costs. Experiments demonstrate improved ranking reliability over FID and reduced variance compared with MIND.

Critical strengths
  1. The paper shows that the entire empirical reference distribution can be represented by interval-averaged quantiles and a residual without changing the sliced Wasserstein score.

  2. The experiments compare multiple feature representations and generative models, demonstrating improvements in runtime and storage.

Critical weaknesses

My main concern is regarding the evaluation:

  1. In Fig.3, the evaluation uses each metric's own large-sample ranking as "ground truth". It does not establish whether QID better reflects actual generative model quality, since different metrics may produce different large-sample rankings.

  2. In Fig.3, it does not demonstrate a consistent advantage of the sample efficiency over Polynomial MMD and RBF MMD. MMD-based methods achieve lower model-selection error.

  3. The comparisons are not consistent. Fig. 3, 6 mainly compare with FID, MMD, while Fig. 4, 5 mainly compare with MIND. Given that QID and MIND achieve nearly identical ranking performance, comparisons with additional baselines would be helpful to establish whether QID offers meaningful advantages.

Questions for authors
  1. What is the comparison of the computational and storage efficiency of QID with MIND? Both methods use the sliced Wasserstein distance and have similar per-batch computational complexity.

  2. Tab.2 says it uses CPU covariance decomposition for FID; will that add extra computational overhead? Does it affect the claim that speedups reach 234× over FID?

5 comments
AuthorsOpenReview ↗

Response to 25HU (part 1)

Thank you for your careful questions about evaluation and computation. QID's transport embedding represents generated batches by projected quantile displacements from the real data. For fixed directions, our finite coordinates preserve the full-reference score as a squared norm, with dimension determined by batch size. At 32 images, QID's mean agreement with human judgments exceeds MIND's in DINOv2 and CLIP, while both kernels have higher mean agreement at 100 directions.

W1. Agreement with human judgments. You are right that recovering a metric's own ranking cannot establish which ordering better reflects realism. We therefore use Stein et al.'s published human judgments [1] as an external criterion. Participants distinguished real from generated images; higher error indicates greater realism.

With DINOv2 at 32 images and 100 directions, QID exceeds MIND on all four datasets, with positive paired 95% intervals and mean gain 0.042 (0.034–0.044). Its selections have 0.71 percentage points higher mean human error than MIND's (0.53–0.89). Both kernels have higher mean correlation and selected-model outcomes. QID's correlation gain over MIND persists at 3,000 directions: 0.020 (0.015–0.022). Table 2 and Appendix O give every dataset for Inception and DINOv2.

We added comparisons of all six metrics for all 41 configurations across ImageNet, CIFAR-10, FFHQ and LSUN Bedroom. We used 50,000-image pools per configuration and reproduced the published means and standard errors from 1,036 participant records. The table gives equal-weight, four-dataset mean Spearman correlations between negative scores and human error, averaged over trials, banks and splits. M counts directions (Tables 36, 45, 52).

Features; images; MQIDfQIDMINDFréchetPolynomial MMDRBF MMD
DINOv2; 32; 1000.7120.7190.6700.4050.7520.752
DINOv2; 32; 3,0000.7470.7490.7270.4050.7520.752
DINOv2; 512; 1000.7760.7740.7740.7750.7850.778
CLIP; 32; 1000.6920.7030.6570.5090.7360.736
CLIP; 32; 3,0000.7340.7400.7040.5090.7360.736

Because correlation assesses the full ordering, a higher value need not imply a more realistic selected model. With DINOv2 and 512 images, QID's selections have 0.51 points higher mean human error than RBF's (0.28–0.75), mainly from FFHQ. The polynomial-MMD difference is 0.10 points (−0.03 to 0.22; Appendix T). QID minus RBF mean correlation is −0.002 (−0.013 to 0.003), and QID minus polynomial MMD is −0.009 (−0.022 to −0.005).

The feature representation changes which models the metrics favor. In all six ImageNet bank–split settings, QID and the kernels share the Inception winner StyleGAN-XL, with the panel's lowest observed human error (15.34%). Their DINOv2 winner, DiT-XL/2 at guidance 4.0, has the highest (32.96%). Actual 512-image Inception selections average 20.42% human error for QID, 16.24% for polynomial MMD and 21.21% for RBF across banks and splits, all below uniform choice (23.98%; Appendix T). Thus reliable recovery, including the FID comparison, need not mean better human realism.

Confirmation with CLIP. With CLIP at 32 images, QID's mean correlation gain over MIND is 0.035 (95% CI 0.030–0.039) at 100 directions and 0.031 (0.025–0.033) at 3,000. We repeated the comparison with OpenCLIP ViT-L/14 to test dependence on DINOv2. At both budgets, paired intervals favor QID on ImageNet, CIFAR-10 and FFHQ; LSUN's include zero. At 100 directions, QID's selections have 0.77 points higher mean human error than MIND's (0.61–0.94), while both kernels have higher mean correlation. At 3,000 directions, QID's mean correlation differences from both kernels have intervals including zero; RBF's selections have 0.16 points higher human error than QID's (0.02–0.30; Appendix U).

AuthorsOpenReview ↗

Response to 25HU (part 2)

W2. Kernel comparisons and winner selection. We agree that polynomial MMD is more reliable for Inception winner selection. Reversing the leading pair can change the winner while leaving other pairs correct. Our added leading-pair analysis shows stronger separation for polynomial MMD relative to sampling variation (Appendix G.4). At 512 images its error is 5.5%, versus QID's 31.8%. RBF recovers more pairs than QID but has higher selection error (44.3%). DINOv2 winner errors are 1.4% for QID, 0% for both kernels and 6.6% for MIND (Table 6); QID has no observed errors at 300 or more directions (Table 47).

QID exceeds both kernels in pairwise agreement across all six Inception bank–split settings at 3,000 directions and 512 images. Primary agreement rises from 93.54% to 95.48%, above the kernels' 94.67% and 94.90%. All six polynomial-MMD and five RBF-MMD paired intervals are positive. Polynomial MMD retains lower winner error: QID's is 29.10% (Appendix R.4).

[1] Stein et al. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. NeurIPS, 2023.

W2, continued: direction budget. Human agreement improves most at the smallest tested batch size when we increase M. With DINOv2, ImageNet correlation rises from 0.714 to 0.782 at 32 images, changes little at 512, and falls from 0.833 to 0.806 on full pools (both kernels: 0.809). We increased M to test the effect of projection sampling on human agreement. The four-dataset mean kernel advantage at 32 images narrows to about 0.006 at 3,000 directions, with paired intervals favoring both kernels (Table 46). RBF's mean advantage persists across the bandwidth check (Appendix S).

W3. Baseline consistency. We agree that varying the baselines obscured the comparison. Across metrics, raw variance depends on score scale, so we compare rankings, selection and human agreement. We completed the six-metric comparisons for ImageNet, COCO Inception, runtime, occlusion and class proportions. At 1,000 images, QID and MIND with 100 directions reach class-proportion correlations of 0.9788 and 0.9364, versus 0.9915 for polynomial MMD (Table 16). Figure 5 holds the sliced score and directions fixed to isolate reference sampling: most of the reduction in variance of score differences comes from reuse, including reuse of a small reference.

AuthorsOpenReview ↗

Response to 25HU (part 3)

Q1. Computation and storage relative to MIND. QID compares an n-image generated batch with all N real images using O(Mnd+Mn log n) online work, as in MIND's comparison of two n-image batches. Its quantile averages and residual preserve the full-reference score using O(Md+Mn) storage after preparation. At N=100,000, n=1,000 and M=100, their measured scoring times are comparable:

FeaturesQID time (ms)MIND time (ms)QID reference (MiB)MIND reference (MiB)
Inception0.7860.8021.177.81
DINOv20.6270.5910.7763.91

QID compares with all 100,000 images. Caching MIND's real projections can give similarly small storage but still targets the 1,000-image real batch. Including directions, QID's storage is 66–99 times smaller than sorted projections with prefix sums for that score. Inception QID uses 1.17 MiB, versus 16 MiB for FID and 781 MiB for full-reference kernels (Table 3). We timed MIND's published function with direction generation and a supplied-direction control (Appendix M).

The larger reference improved human agreement without increasing the generated batch. On ImageNet with DINOv2 and 32 generated images, enlarging the reference from 32 images—the size MIND uses here—to the full set raises human correlation from 0.656 to 0.723 and selected-model human error from 29.28% to 30.37%. Paired gains are 0.067 (95% CI 0.036–0.098) and 1.09 points (0.19–2.19). We added this within-QID reference-size comparison, crossing 32 reference draws with 32 generated trials (Appendix T.2).

Known class proportions let us assess model choices by their total-variation error. In the prespecified paired condition (5% of class mass, 2,000-image batches), QID's loss averaged over banks is 13.1% and 18.1% below MIND's at 100 directions in Inception and DINOv2, and 21.6% and 31.0% below at 3,000. Every bank favors QID at both budgets. We adapted Naeem et al.'s (2020) class-depletion experiment, measuring the selected candidate's total-variation error in known class proportions. At 100 directions QID also improves on reused uniform and class-balanced 2,000-image references. At 3,000, Fréchet has lower loss in both feature spaces, as do RBF in Inception and polynomial MMD in DINOv2. With independently sampled candidates, the QID–MIND intervals at 3,000 directions include zero in both representations (Appendices Q–R).

QID retains the full real benchmark at about the scoring cost of polynomial MMD using a small real sample. For 32-image ImageNet batches with DINOv2 features, both score in about 0.43 ms, using 100,000 real images for QID and 32–512 for polynomial MMD.

Against full-reference kernels at 32 images and 3,000 directions, QID approaches their human agreement at much lower scoring cost. On ImageNet with DINOv2, QID takes 0.420 ms with 100 directions. With 3,000, its human correlation is 0.782 at 0.514 ms, versus 0.778 at 2.504 ms for polynomial MMD and 0.789 at 5.478 ms for RBF MMD (Table 43). At 512 Inception images and 3,000 directions, QID takes 1.52 ms, versus 19.34 and 26.31 ms (Appendix R). Prepared-reference 100-subset KID, reusing real terms and subset indices, takes 10.9–103 ms at 512 and 2,000 generated images in both feature spaces, versus 0.51–1.20 ms for QID (Appendix M.5). Timings start from extracted features and prepared references.

Q2. FID decomposition and the 234× claim. Yes, the original implementation overstated the speedup relative to optimized FID. The revised comparisons replace 234× with 23–43× at 1,000–5,000 images. The original implementation sent the MPS SVD to the CPU and computed singular vectors unnecessary for the score. We therefore rebenchmarked with values-only decomposition, FastFID's smaller Gram formulation [2], and an accelerated GPU/CPU hybrid. All retain the full covariance comparison and pass independent double-precision checks; timings include transfers, synchronization and remaining CPU work. At 32 DINOv2 images, CPU FastFID is faster than GPU QID (0.238 versus 0.420 ms; Appendices M and R.1).

[2] Mathiasen and Hvilshøj. Backpropagating through Fréchet Inception Distance. 2021.

Reviewer_Ho2yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose QID, which evaluates generating models against real datasets. The high level idea is to first embed the synethetic/real images, then use random projections to reduce the dimension, then sort and obtain quantiles and compare using sliced wasserstein distances. The experiments show promising results.

Critical strengths
  • Clarity and Relevance: the paper is written very clearly. It would be easy to follow for anyone from the ICLR community. The topic of generative model/synthetic data evaluation is also highly relevant.

  • Soundness: I find the mathematical development to be straightforward (a compliment) and sound. The empirical studies is reasonably comprehensive, and demonstrates the efficacy of the proposed approach.

Critical weaknesses
  • Insufficient attribution to prior work: There is a large literature on this topic with somewhat related/similar ideas. The current literature review is quite thin. I believe doing a more thorough literature review is crucial to better situate the article within the literature. Note that I am NOT asking the authors to compare with additional competitors in the experiment. I give some example citations that are relevant in the questions section below.

  • This is not a "critical" weakness, but I do want to mention that, perhaps as a byproduct of the simplicity/straightforwardness of the proposed approach (this is a strength/compliment!!), the methodological innovation in the manuscript is somewhat limited, in that much of the elements used in QID, from sliced Wasserstein to random projects etc, are existing, well-known objects, and the contribution conceptually is mostly to combine them for generative model evaluation. On the other hand, I do believe the simplicity of the approach could lead to easier/broader adoption/higher impact.

Questions for authors
  • There should be a more thorough literature review. There are many related work on comparing/evaluating generative models. Some examples are:

  • Distributional Evaluation of Generative Models via Relative Density Ratio. Xu, Wei, and Ma.

  • Assessing the Distributional Fidelity of Synthetic Chest X-rays using the Embedded Characteristic Score. Tam and Engelhardt.

  • Model-Free Assessment of Simulator Fidelity via Quantile Curves. Lin, Wang, Iyengar.

  • I am happy to increase my score if a more comprehensive literature review can be conducted.

2 comments
AuthorsOpenReview ↗

Response to Ho2y

Thank you for recommending these papers and for recognizing simplicity as a strength. QID addresses repeated evaluation of small generated batches against a much larger real dataset. You are right that its established ingredients need fuller attribution and its contribution clearer placement. We have expanded Section 6 and Appendix N with discussion of 18 methodological and evaluation papers, including your three suggestions, explaining their evaluation goals and how QID builds on the transport literature.

Statistical comparison and distributional diagnosis. Xu, Wei and Ma [1] connect aggregate model comparison with diagnosis of individual observations and covariates through their relative density-ratio approach. They learn a bounded function from real and generated samples, while QID prepares its reusable reference coordinates from real data alone. We now discuss this relationship alongside classifier two-sample tests, which assess distinguishability on held-out samples, and relative-MMD tests, which compare generators while accounting for their shared real sample.

Tam and Engelhardt's Embedded Characteristic Score [2] is relevant to QID's aim of detecting differences beyond Gaussian moments. We now explain how ECS uses characteristic functions of feature coordinates, whereas QID uses projected empirical quantiles, and how both use variation among real samples to interpret score magnitudes. Their chest-X-ray study also illustrates the importance of domain-relevant features. We discuss these approaches alongside learned spatial and frequency features for two-sample testing, density and coverage, and authenticity measures for training-data copying.

Uncertainty across evaluation settings. Iyengar, Lin and Wang [3] use confidence sets to account for finite observations within each scenario when estimating quantile profiles of simulator discrepancies. Their scenario-level view complements our study of repeated generative evaluation: it describes how discrepancies vary across evaluation settings, while QID's quantiles describe projected features within a distribution comparison.

Contribution. We designed QID so that using more real data does not make each subsequent evaluation more expensive. We build on optimal transport embeddings to represent generated distributions relative to the real benchmark, giving their comparisons a shared geometry. We then show that the reference information needed by every generated batch of a chosen size can be prepared once in a compact form. For fixed projection directions and equally weighted batches, these coordinates preserve both the full-reference sliced-Wasserstein score and transport comparisons between batches. This lets researchers reuse the entire real benchmark across generators, while reference storage and scoring depend on the generated batch size (Proposition 3.1; Appendix B). The benefit extends beyond faster computation. Small generated batches can use all the available real data, and the reference can be distributed and reused across evaluations.

With 100,000 real images and batches of 1,000, QID uses 66–99 times less storage than sorted projections with prefix sums for the full-reference score, with comparable scoring time (Table 3). The reference can therefore be prepared once and reused throughout model evaluation in this compact form. In Section 6 and Appendix N, we explain how these interval averages and residual differ from PSWE's reference-grid interpolation for unequal set sizes and FourierSWE's cosine coefficients.

With 32 generated images, QID's mean agreement with published human judgments exceeds MIND's with DINOv2 and CLIP features (Section 5.2; Appendices O and U). This comparison covers 41 generator configurations. The expanded review connects this human study to its HYPE evaluation protocol and our controlled experiments to prior empirical metric comparisons. We will publicly release the implementation and prepared reference files after review, with examples for preparing a reference and reusing it across evaluations.

We hope the expanded review addresses your concern, and we would gladly discuss any further work you consider relevant.

[1] Xu, Wei and Ma. Distributional Evaluation of Generative Models via Relative Density Ratio.

[2] Tam and Engelhardt. Assessing the Distributional Fidelity of Synthetic Chest X-rays using the Embedded Characteristic Score.

[3] Iyengar, Lin and Wang. Model-Free Assessment of Simulator Fidelity via Quantile Curves.

Dear reviewer,

Since the discussion period closes today, we wanted to follow up briefly. We would be grateful to hear whether the expanded related work (Section 6 and Appendix N, including all three papers you suggested) addresses your concern. Thank you again for your constructive feedback and very kind and thoughtful review.

Best regards, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

To all reviewers and the AC

We thank the reviewers for their careful feedback. We've uploaded a revised draft of the paper to OpenReview and summarize our changes here - in addition to responding to each reviewer individually below.

In response to the reviews, we have substantially expanded the experiments and literature review. The revised study now spans six metrics, five datasets and over two million real and generated images, using Inception, DINOv2 and CLIP features. QID achieves substantial scoring speedups, evaluating generated features up to 95× faster than full-reference kernel estimators and 43× faster than optimized FID. The additions address the reviewers' questions about model quality, small samples, kernel comparisons and computational cost.

  • Human agreement and CLIP confirmation. The reviewers asked whether reliable rankings also reflect model quality. We added comparisons of all six metrics with published human judgments for 41 configurations across four datasets, evaluating rankings and the realism of selected models. We then repeated the small-batch comparison with CLIP to test dependence on the feature representation. At 32 generated images and 100 directions, QID's mean correlation gains over MIND are 0.042 with DINOv2 and 0.035 with CLIP, both statistically significant. Both kernels have higher mean agreement at this budget (Section 5.2; Appendices O and U).

  • Very small batches and the value of real data. To address the two small-sample regimes, we added experiments with 1–32 generated images and 8–512 real-reference images, varying the sizes separately. We also evaluated the reference-size comparisons against human judgments. On ImageNet with DINOv2 and 32 generated images, increasing the real reference from 32 images to the full set raises correlation from 0.656 to 0.723. Additional controlled class-frequency experiments test the benefit of retaining reference information when the candidate errors are known (Appendices P, Q and T).

  • Kernel comparisons and model selection. The reviewers asked why polynomial MMD selects the winner more reliably and why the baseline sets varied. We added a leading-pair analysis and completed six-metric comparisons on ImageNet, COCO Inception, occlusion and class-proportion tests. We also studied 100–3,000 directions to assess the effect of projection sampling. At 512 generated images and 3,000 directions, QID recovers more model pairs than both kernels across all three projection banks and both data splits in Inception space. Polynomial MMD retains lower winner-selection error, with stronger separation of the leading models relative to sampling variation. We also added an RBF bandwidth study (Appendices G.4, R and S).

  • Substantial savings with optimized baselines. The reviewers questioned whether CPU decomposition inflated the FID comparison. We rebenchmarked against optimized full-covariance FID, MIND's published function and prepared-reference 100-subset KID. QID scores 23–43× faster than optimized FID at 1,000–5,000 generated images. We also compare representations that preserve the full-reference score: with 100,000 real images and batches of 1,000, QID uses 66–99 times less storage than sorted projections with prefix sums, with comparable scoring time (Table 3; Appendix M).

  • Expanded attribution and explanation. To address the request for a broader literature review, we expanded Section 6 and Appendix N with 18 additional methodological and evaluation papers, including all three suggested works. We explain how QID builds on transport embeddings and relates to statistical model comparison, distributional diagnostics and uncertainty in evaluation, clarifying the role of its finite representation.

Our individual responses explain the findings relevant to each concern. We will publicly release the implementation and prepared reference files after review, with examples for preparing a reference and reusing it across evaluations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.