Multiscale Generative Modeling by Recursive Refinement

Physical systems exhibit structure across spatial scales, from large-scale organization to fine details. Coarse representations describe these systems with fewer degrees of freedom but do not uniquely determine their fine-scale structure. Learning the distribution of these unresolved structures allows generation to proceed through smaller refinement problems, each conditioned on the coarse structure already established. Here, we introduce a discrete multiscale generative framework that constructs samples through recursive refinement directly on the native grid, without a learned encoder or decoder. A single model is shared across scales and learns the conditional distribution of fine-scale configurations, with an explicit spatial correspondence between coarse and fine grids. The framework supports different stochastic refinement processes, including spin dynamics for binary structures and mass-conserving dynamics for particle allocation. The conservative processes preserve particle counts exactly, ensuring that every generated refinement aggregates to its prescribed coarse state. We demonstrate the framework on natural images, Julia-sets, and three-dimensional porous media. For Julia-sets, recursive refinement extends to scales unseen during training. For porous media, the model generates volumes of $1024^3$ voxels with exactly prescribed porosity (void fraction), with no fixed architectural limit on volume size. We show that a single refinement model can generate structure across scales while honoring coarse-scale constraints.