Reducing Hallucinations in Generative Models through Truncated Statistics

Hallucinations—where generative models produce invalid or nonsensical outputs—remain a critical challenge for reliable deployment. We present the first computationally and query-efficient algorithm that provably addresses the hallucination problem by actively querying the model’s own invalid outputs. Specifically, we control the hallucination rate up to an arbitrarily small multiplicative slack while approximately maximizing the likelihood of valid target examples via projected stochastic gradient descent. Our method works in very general settings with arbitrary validity sets and valid observations, within fixed powerful exponential families satisfying stated regularity and sampling assumptions. Our approach is enabled by a novel connection to the field of truncated statistics and settles an open problem posed by Hanneke et al. (2018).