Your VAE Knows Where to Sample, Not What It Doesn't Know: Merged Aggregated Posteriors as Deterministic Priors and Cluster Maps

The aggregated posterior of a trained VAE, the $N$-component Gaussian mixture of its per-sample posteriors, can be compressed to $K$ components by moment-preserving hierarchical merging: one deterministic pass, with no seed and no re-fitting, yields the entire nested family, every $K$ from $N$ to $1$. We evaluate this object in three roles. As a latent density for out-of-distribution detection it fails: a pre-registered study (MNIST, CIFAR-10; with CelebA-64 added later) refutes it in every form we could construct — post-hoc, co-trained, IWAE-trained, as a discrete codebook, substituted into the pixel likelihood, and at every $K$ along the path — across three dataset regimes. On CelebA-64, latent density sits near or below chance against SVHN, but pixel based density (with a calibrated decoder) is even worse. The easy-regime (MNIST) verdict turns on the training recipe, not on the object: ordinary geometric augmentation with a convolutional architecture makes MNIST latent density beat that same model's pixel likelihood at $\beta =0.5$ ($10/10$ seeds), while the CIFAR-10/SVHN inversion barely moves. As a structure recovery device it succeeds: co-evolving the merge partition during training raises the clustering ceiling decisively over the best plain pipeline (NMI $0.74$ vs. $0.66$; plain ELBO plus a KL floor suffices), though not to the $0.86$ of DEC, which gives up the generative model. A controlled comparison shows the hierarchy itself carries this structure ($+0.12$ NMI over post-hoc merging on the same encoder), while density depends only on the encoder ($\Delta \le 0.008$ AUROC on MNIST). As a generative prior it succeeds too: drop-in sampling beats $\mathcal{N}(0,I)$ on $10/10$ seeds on every dataset, a frozen merged codebook beats learned codebooks in an SQ-VAE-style model, and, unlike an equal-$K$ EM prior, the mixture cannot catastrophically corrupt the model's likelihood. Against EM, which samples as well, the merge's advantage is this safety, determinism and nesting, not accuracy. One principle organizes all three verdicts: moment-preserving compression preserves where the data lives, its support and cluster structure, not how likely it is.