Why Diffusion Sampling Can Ignore Ambient Dimension: Rate-Distortion Bounds in KL and TV

Diffusion models sample high-dimensional data in few steps. Yet existing discretization bounds scale with an intrinsic dimension of the data or, for Gaussian mixtures, with the squared logarithm of the number of components. We show that what matters instead is the information the data carry at the target resolution. For the posterior-mean exponential integrator with $K$ steps, the time-discretization error in total variation is $O(I_{\Gamma }L/K)$. Here $I_{\Gamma }$ is the mutual information between the data and their Gaussian observation at the terminal signal-to-noise ratio, and $L$ is the log-SNR range. The bound is linear in $L/K$, whereas Pinsker's inequality applied to information-theoretic KL bounds gives only $\sqrt{L/K}$. We bound $I_{\Gamma }$ by the rate-distortion function of the data. Up to a constant, it is at most the metric entropy of the support, and for Gaussian mixtures it is at most the label entropy, so our bound improves on both. The implication: as far as discretization error is concerned, diffusion in pixel space does as well as diffusion in latent space, whatever the number of pixels.