The Sample Cost of Accurate Model Comparison in Total Variation
Abstract
Choosing between two generative models asks for one decision rather than an estimate of either model's error. How much data can this distinction save? We study two completely known distributions on an alphabet of size , with independent evaluation data from an unknown target distribution. A classical selector achieves a factor-3 approximation to the smaller total variation distance using a number of samples independent of . We show that every fixed improvement below factor 3 requires samples in the worst case, at sufficiently small fixed additive error. The known upper bound for estimating total variation gives a matching dependence on . Thus accurate comparison and numerical distance estimation have the same domain-size complexity, even with only two simple candidate distributions. The proof preserves a reversed model ranking while matching logarithmically many moments of the target probabilities. We give a general moment construction, a direct Poisson-thinning argument, and an explicit reduction between exact-factor comparison and distance estimation.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.