When is Compositional Generation Feasible? Distributional Estimation Error and Inference-time Approximation Error in Diffusion Models

The task of compositional generation involves using a conditional generative model, trained only on a subset of the possible conditions, to produce samples from compositionally-defined target distributions such as a geometric combination of the source distributions. In this work, we explore what makes this challenging by separating out several sources of error and analyzing their interplay. To begin, we show that even when source distributions are estimated with arbitrarily small error, their composition may be arbitrarily far from the true composition. This leads us to derive sufficient conditions to prevent error blowup, and we propose a metric based on the resulting error-sensitivity bounds which can separate error-sensitive compositions from stable ones without access to ground truth distributions. Then, we carefully design experiments with diffusion models for both synthetic and realistic data to analyze the interaction between distributional estimation error and inference-time approximation error. In particular, while recent methods can reduce inference-time approximation error in principle, our results show that distributional estimation error has a more catastrophic effect on performance when the target distribution is out-of-distribution with respect to the sources.