- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies the sample complexity of estimating the Frechet Inception Distance, which is widely used in generative modeling settings as a measure of discrepancy between probability distributions. The first contribution is to show that the standard plug-in estimator for the FID has a sample complexity whose leading order term is , which arises from the bias. The bias is controlled by representing the empirical covariance as a random perturbation of the population covariance and controlling the term in the error of the plug-in estimator which is nonlinear in using its resolvent representation. The authors then study another estimator, "FID-VALE", obtained by dividing the data into independent batches, taking a convex combination of plug-in estimates for the FID score using samples from each batch, and choosing the coefficients in such a way that the higher-order terms in the bias cancel out; this is a generalization of the . They prove canceling the first terms in the bias yields an estimator with sample complexity. Finally, the authors introduce a third estimator, "FID-RTD", which arises from directly estimating higher-order terms in the expansion of the nonlinear-in- term in the plug-in estimator. FID-RTD achieves a sample complexity of , which is complemented by an information-theoretic lower bound of the same order. Experiments on both Gaussian distributions and image datasets illustrate the dimensional scaling of their estimates and the improvement yielded by FID-VALE and FID-RTD.
Critical strengths
FID is used ubiquitously in the evaluation of learned generative models, and any work that improves our understanding of its finite-sample behavior and informs the design of more statistically-efficient estimation methods is of high value to the ML community. The experimental results on FID-VALE and FID-RTD suggest that the methods introduced in this paper can be of immediate use in the evaluation of generative models.
The paper addresses a notable gap in the theory literature, the results are quite impressive, and I found the proof techniques to be quite elegant. I also applaud the authors for writing such a clear exposition of their results.
Critical weaknesses
Since the analysis is specialized to the BW distance, it seems unlikely that the techniques introduced in this work can be applied more broadly to study the sample complexity of other discrepancies which depend on higher-order moments. However, this is quite a minor weakness, if one at all.
Questions for authors
I am curious whether the dimensional-dependence of the sample complexity of the various estimators analyzed in the paper can be improved if the covariances and are assumed to possess some low-complexity structure (e.g., approximately low-rank covariances, distributions supported on a low-dimensional manifold, sparse graphical models, distributions which are invariant under a group action, etc). This seems like a challenging direction, since your proofs involve controlling the bias and variance of highly nonlinear functions of Wishart matrices, but I am curious nonetheless.
Minor comments:
- I was confused by the following sentence in Sec 1.1: "This setting is particularly natural for FID, which by its very definition as a squared Wasserstein distance between moment-matched Gaussians, assumes the image embeddings are close to a Gaussian (else measuring this distance is ill-posed)." Does this simply mean that the FID score does not define a metric between non-Gaussian distributions?
- In Problem 1, it may be worth to briefly comment on why you focus on the 1-sample problem.