Fixing the Fréchet Inception Distance

The Fr´echet Inception Distance (FID) is the staple metric to assess the progress of image generation models. In practice, its evaluation requires a large number of samples (n ≥ 50k, both real and generated) to be accurate. Hence, the FID is not suitable for scenarios involving a small number of samples, which are common in specialized domains. In this work, we introduce FxD: a strictly compatible replacement to the usual closed-form of the FID. We replace the classical estimator of the Bures-Wasserstein distance between the two population covariance matrices by a variant from the random matrix theory. We show that the classical FID is still biased under the standard evaluation protocol (n = 50k) due to its slow convergence. In contrast, FxD converges quickly to the desired quantity, with sample sizes as low as a few thousands images. We demonstrate the interest of FxD for fast and frugal evaluation of generative models both on synthetic datasets and real-world datasets at various scales, including ImageNet-1k, CUB200, Oxford102 and two specialized datasets from medical imaging and remote sensing.