- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The Quantile Inception Distance (QID) method introduces a new evaluation metric for generative models. It is based on the concept of optimal transport embedding (using the Sliced Wasserstein distance, ) to project and compare feature distributions against a common real-world reference dataset. This method aims to compare generated images with original ones in particular in the low generated sample framework. Its main advantage lies in its ability to take into account higher order moments (not only mean and covariance like the Fréchet Inception Distance (FID)). This allows to detect subtle changes in the shape of distributions. Experiments have been done on Imagenet, COCO and CIFAR-10 databases. The results show improve in the pairwise ranking error of the chosen model.
Critical strengths
1- the evaluation is performed in the generated "low sample size" framework and proves interest. 2- The offline computation of the feature map on the real data set enables to have an efficient online algorithm for each new generated sample. 3- The Gaussian approximation increases this computational efficience without a significant deterioration of the results. 4- The theoretical foundations of this methods make it interesting.
Critical weaknesses
1- Although the method seems to perform better than FID, Figure 3 shows that on Imagenet, Polynomial MMD is more performant in partiuclar for the selection error. Can you explain this behaviour? 2- the authors mention the "low generated sample size" framework but are still working with several hundreds of generated samples. What would happen in a lower regime?
Questions for authors
In addition to the questions related to weaknesses, I would really appreciate if the authors can explain what would happen in the opposite regime: very low sample size as reference and large (or not necessarily) generated datasets.
3 comments
Response to TUr7 (part 2)
What if the real reference is small? More generated images refine the generated quantiles while leaving the real-reference quantiles unchanged. At N=8,n=5000, Inception QID recovers 94.8% of the small-reference ordering but only 59.4% of the full-reference ordering. At N=512,n=5000, full-reference agreement reaches 96.5% in Inception and 99.4% in DINOv2. Both kernels recover more full-reference pairs throughout N≤128. To separate the effects of real and generated sample sizes, we added experiments varying N=8,32,128,512 and n=32,1000,5000 and comparing recovery of the small-reference and full-reference orderings. We cross 32 reference draws with 32 generated trials (Appendix P.2). Intervals resample the reference and generated-trial axes separately.
The value of additional real data for selected-model realism depends on the feature representation. With DINOv2 and 5,000 generated images, enlarging the reference from N=8 to the full set raises QID's selected-model human error from 29.77% to 32.96%: +3.19 points (95% CI 1.07–5.61). Inception correlations remain negative; enlarging its reference lowers selected-model human error from 22.97% to 15.75%. We evaluate this reference-size grid against human judgments (Appendix T). At N=512,n=5000, QID and both kernels reproduce their full-reference DINOv2 choices in every observed trial. DINOv2 kernels remain stronger at N=8.
QID's representation holds for every positive N and n. With n>N, storing n coefficients per direction saves no space over N sorted reference values. When real data are plentiful, the averages and residual preserve the full-reference comparison in space proportional to the generated batch.
Dear reviewer TUr7,
With the discussion window ending today, we wanted to check in on your questions about very small samples. The revision adds experiments down to a single generated image (Appendix P.1), real references as small as 8 images for the opposite regime you asked about (Appendix P.2), and an analysis of why polynomial MMD selects the winner more reliably (Appendix G.4).
We hope these answer your questions and would welcome any final thoughts. We appreciate the care you took with the paper.
Kind regards, authors
Response to TUr7 (part 1)
Thank you for your questions and and encouraging review. We are happy to continue the conversation below if the reviewer has any further or follow-up questions.
Why does polynomial MMD select the winner more reliably? Polynomial MMD's score gap between the leading models is larger relative to sampling variation (Figure 3). Reversing that pair can choose the wrong winner while leaving 54 of 55 pairs correct. At n=512 in Inception, this pair reverses in 157 of QID's 163 selection errors. We added a score-gap analysis of the common winner and runner-up, StyleGAN-XL and DiT-XL/2 with guidance 1.5. Their mean gap is 1.43 sampling standard deviations for polynomial MMD and 0.49 for QID. They recover this pair's ordering in 94.7% and 69.3% of trials, with selection errors of 5.5% and 31.8%. DINOv2 also gives polynomial MMD stronger leading-pair separation (Appendix G.4). At 3,000 directions, QID's primary Inception agreement rises to 95.5%, but selection error remains 29.1% (Appendix R.4).
Feature choice changes the realism of the models these metrics favor. In all six ImageNet bank–split settings, the common Inception winner has the panel's lowest observed human error (15.34%); the DINOv2 winner has the highest (32.96%). Higher error means real and generated images are harder to tell apart. We also evaluate 512-image selections against human judgments (Appendix T), averaging banks and splits. QID, polynomial MMD and RBF MMD selections have human errors of 20.42%, 16.24% and 21.21% in Inception, all below uniform choice (23.98%), and 32.95%, 32.96% and 32.96% in DINOv2.
What happens below hundreds of generated images? QID can still use the full real reference when few generated images are available. It exceeds MIND and Fréchet distance at n=8,16,32 in every bank and split; both kernels have higher agreement. At n=32, QID recovers 72.1% of pairs in Inception and 85.3% in DINOv2, exceeding MIND by 6.57 and 4.74 percentage points. To test whether the gains persist as fewer observations determine the generated quantiles, we added experiments at n=1,8,16,32,64 on all eleven ImageNet configurations, with 512 paired trials, both encoders, three banks and two splits (Appendix P.1).
With DINOv2 features, QID's small-batch rankings agree more closely with human judgments than MIND's. We use Stein et al.'s (2023) human judgments and report mean Spearman correlations for the primary bank and split below (Appendices O–P).
The 32-image QID–MIND gain remains positive after omitting any configuration in every bank and split. With all configurations, the primary comparison gives 0.088 (95% CI 0.064–0.099). Both kernels remain stronger, and Inception correlations are negative. The mean DINOv2 gain is 0.042 (0.034–0.044) in our expanded comparison of all 41 configurations across four datasets, averaging banks and splits (Table 2, Appendix O.3). With CLIP at 32 images and 100 directions, repeating the comparison gives 0.035 (95% CI 0.030–0.039; Appendix U). On ImageNet with DINOv2 features at 32 images, more directions bring QID close to kernel human agreement at lower scoring cost (Table 43).
The table above shows that fQID ranks these batches well using moments, while QID also detects the change preserving the feature mean and covariance (Table 4). At n=1, the generated quantile is constant and its variance is zero, so QID equals fQID. At this size, ranking recovery is 52.7% in Inception and 47.5% in DINOv2 for the primary bank and split (Appendix P.1, Figure 16). Unbiased covariance and kernel estimates require at least two observations and are omitted.