acceptodds
Under review as a conference paper at ICLR 2027

Fixing the Fréchet Inception Distance

Abstract

The Fr´echet Inception Distance (FID) is the staple metric to assess the progress of image generation models. In practice, its evaluation requires a large number of samples (n ≥ 50k, both real and generated) to be accurate. Hence, the FID is not suitable for scenarios involving a small number of samples, which are common in specialized domains. In this work, we introduce FxD: a strictly compatible replacement to the usual closed-form of the FID. We replace the classical estimator of the Bures-Wasserstein distance between the two population covariance matrices by a variant from the random matrix theory. We show that the classical FID is still biased under the standard evaluation protocol (n = 50k) due to its slow convergence. In contrast, FxD converges quickly to the desired quantity, with sample sizes as low as a few thousands images. We demonstrate the interest of FxD for fast and frugal evaluation of generative models both on synthetic datasets and real-world datasets at various scales, including ImageNet-1k, CUB200, Oxford102 and two specialized datasets from medical imaging and remote sensing.

Primary area:
generative models
Keywords:
FID, evaluation, metric, generative models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rGSyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper questions the commonly used "FID@50k" protocol for FID. It points out that the classic plug-in estimator is far from converged when the feature dimension is p = 2048 and the sample size is only tens of thousands, causing it to systematically overestimate the true Fréchet distance, and making it especially unreliable on low-sample datasets. The authors borrow the direct estimator of the Bures-Wasserstein distance from random matrix theory (Tiomoko & Couillet, 2019) to construct FxD: they replace the plug-in covariance term in classic FID with an RMT-corrected covariance distance, thereby obtaining faster convergence in an asymptotic framework where n and p are of the same order and p/n → c ∈ (0, 1). Experiments cover synthetic Gaussians, BigGAN/DiT on ImageNet, small datasets CUB200/Flowers102, breast MRI, and remote sensing RSCID, showing that classic FID still clearly overestimates at 50k samples, while FID∞ and FxD converge to similar and lower values.

Critical strengths

The problem is important and the empirical findings are striking. FID is one of the most commonly used metrics for generative model evaluation. The paper uses BigGAN/DiT on ImageNet to show that FID@50k is still far from converged and that the true Fréchet distance is about 20% lower, and uses FID∞ and FxD to corroborate each other. This is not merely a theoretical concern but an empirical challenge to the community's default protocol, with direct methodological value.

FxD is simple, low-overhead, and evaluated broadly. The estimator implementation requires only two additional matrix decompositions compared with standard FID, and its runtime is much lower than FID_∞. Experiments cover synthetic Gaussians, large models, small datasets, medical imaging, and remote sensing, and the code is open-sourced. If the authors can clarify protocol alignment and the n < p case, FxD has the potential to become a practical alternative for low-sample FID evaluation.

Critical weaknesses

Fair comparison under the standard asymmetric protocol and deployment limitations are not sufficiently resolved. Standard FID usually fixes n_r as the full set of real images and only varies n_g, whereas the closed-form version of FxD mainly targets n_r = n_g. On ImageNet, the paper compares FxD with n_r = n_g against classic symmetric/asymmetric FID, but does not directly show FxD's advantage under the standard asymmetric protocol where "n_r is fixed to the full real set and n_g is very small." In practical applications, the real set is often fixed and much larger than the generation budget, and this limitation directly affects FxD's deployability.

There is tension between the theoretical assumptions and the low-sample claim, and the ranking impact is not proven. The RMT theorem requires n > p and an invertible covariance, but the "low-sample" scenario most emphasized in the paper (e.g., RSCID with n = 1093 < p = 2048) violates this assumption; the behavior of FxD in such cases needs theoretical or at least numerical clarification. In addition, the paper proves absolute FID bias but does not prove that model rankings will change, whereas the main purpose of using FID in the community is precisely ranking.

Questions for authors

(1) Can you directly compare FxD with classic FID@50k under the standard asymmetric protocol (with n_r fixed to the full real set and n_g varied from hundreds to 50k)? Can the numerical integration instability of asymmetric FxD be addressed through a stable implementation or approximation scheme? This directly determines whether FxD can replace the standard FID workflow. (2) For settings such as RSCID where n < p, how exactly is FxD computed? Does it use a pseudo-inverse, regularization, or still rely on the theorem's n > p condition? Can you provide a theoretical guarantee or at least empirical calibration for n < p? (3) Can you add model-ranking experiments: compare the ranking consistency of classic FID@50k, FID_∞, and FxD on multiple generators (not just BigGAN/DiT)? The paper acknowledges that bias may not affect rankings, but if rankings also do not change, how should the claim that FID@50k is "broken" be stated more precisely? (4) Real Inception/DINOv2/CLIP features are clearly non-Gaussian. Can you provide bias/variance comparisons between FxD and classic FID under heavy-tailed, low-rank, and non-Gaussian synthetic features? Is the finite fourth-moment assumption verifiable on real features? (5) FxD relies on n_r = n_g or at least the two being of the same order; on practical small datasets, the authors suggest generating as many images as the real set. If the generation budget is much larger than the real set, is FxD still recommended, or should the asymmetric version be used? Please provide a practical decision rule.

Reviewer_bpiDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses the limitations of the FID measure for small data samples by introducing FxD, a random matrix theory-based approach that the authors claim is better suited to smaller datasets. The paper questions the rationale for the arbitrary 50k bound on the FID.
The Tiomoko & Couillet theorem-based strategy, which utilizes an RMT-based distance estimator not bound by sample sizes, is demonstrated to perform effectively for synthetic data with known ground truths Then demonstrate the stability of the RMT distance measure, compared with FID, on standard datasets such as ImageNet for well-established models such as DiT and BigGAN.

Critical strengths

Examines a pertinent problem with the FID, given its wide use as a benchmark for GenAI performance evaluation. It also raises concerns about the arbitrary 50k bound selection. Hence, the research question is well motivated. The experimental validation is well planned and covers synthetic to standard datasets (ImageNet Oxford Flowers, etc.) over well-established generative AI models to demonstrate the stability of the RMT measure. It also demonstrates feasibility for low-sample support. Then expand the experiments to specialized dataset domains.

Critical weaknesses

The novelty of the paper, in essence, is the adaptation of the RMT-based formulation of the FxD metric as a replacement for FID for datasets with smaller sample sizes. Malik Tiomoko and Romain Couillet. Random matrix-improved estimation of the Wasserstein distance between two centered Gaussian distributions. In 2019 27th European Signal Processing Conference (EUSIPCO), pp. 1–5. IEEE, 2019. – The contribution of the work compared to this reference is essentially the adaptation of the method proposed in this paper as a replacement for the FID measure. Hence, the contribution is primarily experimental. While useful, this doesn’t constitute sufficient research novelty for publication in this conference, based on the reviewers' opinion. Furthermore, the stability of the distance measure for the RMT metric is not sufficiently motivated by theoretical explanations. Hence, the observation may not generalize. Finally, the more comparable performance to FID_inf in some datasets remains unexplained. As FID_inf by itself seems to resolve the issue on most occasions to a sufficient degree. Essentially, the theoretical or algorithmic contribution should be highlighted.

Questions for authors

Does this measure capture information from Fidelity metrics such as Density and Coverage? Is there a theoretical explanation for FID_inf's comparable performance to FxD on some datasets? Can you distinguish more clearly the theoretical contribution of your work compared to the work by Malik Tiomoko and Romain Couillet. Random matrix-improved estimation of the Wasserstein distance between two centered Gaussian distributions. In 2019, 27th European Signal Processing Conference (EUSIPCO), pp. 1–5. IEEE, 2019, is that beyond its adaptation for the research problem?

Reviewer_1ubSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The submission revisits the standard FID@50k evaluation protocol and argues it is unreliable. The key observation is that the classical plug-in FID estimator is strongly biased in the low-sample regime and still not converged even at n=50k. As a remedy, an RMT-based estimator of the Bures–Wasserstein distance (adapted from Tiomoko & Couillet, 2019) is introduced under the name FxD, which is claimed to stabilize with roughly an order of magnitude fewer samples. Experiments cover synthetic Gaussians, ImageNet-1k with BigGAN and DiT, small fine-grained datasets (Flowers 102, CUB200), and specialized domains (breast MRI, remote sensing), using Inception v3, DINOv2, and CLIP features. Classical FID is reported to overestimate the underlying distance by about 20% on ImageNet.

Critical strengths
  • The sample-size dependence of FID is widely suspected but rarely quantified. The paper turns this into a concrete, measurable claim, which is useful for the community regardless of whether one adopts FxD.

  • FxD is a closed-form estimator that relaxes the Gaussianity assumption and integrates easily into existing FID pipelines, making it a low-friction alternative to the standard estimator.

  • Experiments span synthetic data, a large-scale benchmark, fine-grained small datasets, and specialized domains, across three feature extractors and multiple generators, with a consistent qualitative trend throughout

Critical weaknesses
  • Convergence to a plateau may not imply convergence to the true Fréchet distance. The argument treats FxD's early plateau as the "actual Fréchet distance" and thus reports FID@50k as "20%" off. This is plausible, but the plateau could also be a fixed residual bias rather than the ground truth. A comparison against the analytically known distance in the synthetic Gaussian setting would directly test whether the plateau is unbiased, and would considerably strengthen the claim.

  • Theorem 1 assumes nr=ng=nn_r=n_g = n with n>pn>p , while the standard protocol uses $n_r=N≠n_g=50k $ and one experiment (RSCID, n=1093 <p=2048) lies outside this regime; some discussion of the nr≠ngn_r≠n_g case and a qualification of the "strictly compatible plug-and-play" claim would be helpful.

  • On ImageNet only two generators (BigGAN and DiT) are evaluated, while several recent families cited in the paper (e.g., flow matching, PixelDiT) are discussed but not tested; a text-to-image dataset in the main results would make the practical recommendation more convincing.

Questions for authors

see the weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.