- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper questions the commonly used "FID@50k" protocol for FID. It points out that the classic plug-in estimator is far from converged when the feature dimension is p = 2048 and the sample size is only tens of thousands, causing it to systematically overestimate the true Fréchet distance, and making it especially unreliable on low-sample datasets. The authors borrow the direct estimator of the Bures-Wasserstein distance from random matrix theory (Tiomoko & Couillet, 2019) to construct FxD: they replace the plug-in covariance term in classic FID with an RMT-corrected covariance distance, thereby obtaining faster convergence in an asymptotic framework where n and p are of the same order and p/n → c ∈ (0, 1). Experiments cover synthetic Gaussians, BigGAN/DiT on ImageNet, small datasets CUB200/Flowers102, breast MRI, and remote sensing RSCID, showing that classic FID still clearly overestimates at 50k samples, while FID∞ and FxD converge to similar and lower values.
Critical strengths
The problem is important and the empirical findings are striking. FID is one of the most commonly used metrics for generative model evaluation. The paper uses BigGAN/DiT on ImageNet to show that FID@50k is still far from converged and that the true Fréchet distance is about 20% lower, and uses FID∞ and FxD to corroborate each other. This is not merely a theoretical concern but an empirical challenge to the community's default protocol, with direct methodological value.
FxD is simple, low-overhead, and evaluated broadly. The estimator implementation requires only two additional matrix decompositions compared with standard FID, and its runtime is much lower than FID_∞. Experiments cover synthetic Gaussians, large models, small datasets, medical imaging, and remote sensing, and the code is open-sourced. If the authors can clarify protocol alignment and the n < p case, FxD has the potential to become a practical alternative for low-sample FID evaluation.
Critical weaknesses
Fair comparison under the standard asymmetric protocol and deployment limitations are not sufficiently resolved. Standard FID usually fixes n_r as the full set of real images and only varies n_g, whereas the closed-form version of FxD mainly targets n_r = n_g. On ImageNet, the paper compares FxD with n_r = n_g against classic symmetric/asymmetric FID, but does not directly show FxD's advantage under the standard asymmetric protocol where "n_r is fixed to the full real set and n_g is very small." In practical applications, the real set is often fixed and much larger than the generation budget, and this limitation directly affects FxD's deployability.
There is tension between the theoretical assumptions and the low-sample claim, and the ranking impact is not proven. The RMT theorem requires n > p and an invertible covariance, but the "low-sample" scenario most emphasized in the paper (e.g., RSCID with n = 1093 < p = 2048) violates this assumption; the behavior of FxD in such cases needs theoretical or at least numerical clarification. In addition, the paper proves absolute FID bias but does not prove that model rankings will change, whereas the main purpose of using FID in the community is precisely ranking.
Questions for authors
(1) Can you directly compare FxD with classic FID@50k under the standard asymmetric protocol (with n_r fixed to the full real set and n_g varied from hundreds to 50k)? Can the numerical integration instability of asymmetric FxD be addressed through a stable implementation or approximation scheme? This directly determines whether FxD can replace the standard FID workflow. (2) For settings such as RSCID where n < p, how exactly is FxD computed? Does it use a pseudo-inverse, regularization, or still rely on the theorem's n > p condition? Can you provide a theoretical guarantee or at least empirical calibration for n < p? (3) Can you add model-ranking experiments: compare the ranking consistency of classic FID@50k, FID_∞, and FxD on multiple generators (not just BigGAN/DiT)? The paper acknowledges that bias may not affect rankings, but if rankings also do not change, how should the claim that FID@50k is "broken" be stated more precisely? (4) Real Inception/DINOv2/CLIP features are clearly non-Gaussian. Can you provide bias/variance comparisons between FxD and classic FID under heavy-tailed, low-rank, and non-Gaussian synthetic features? Is the finite fourth-moment assumption verifiable on real features? (5) FxD relies on n_r = n_g or at least the two being of the same order; on practical small datasets, the authors suggest generating as many images as the real set. If the generation budget is much larger than the real set, is FxD still recommended, or should the asymmetric version be used? Please provide a practical decision rule.