One Rule for Every Scale: Recovering Fine Detail with Self-Similarity

Neural networks modeling natural signals often correctly learn coarse spatial scales but struggle to resolve fine scales, leading to blur, missing detail, and rollout drift. This attenuation of fine detail has been observed independently across many domains, like image reconstruction and neural PDE solvers. We observe that many of these natural signals exhibit approximate self-similarity, meaning that their fine scales are largely predictable from their coarse scales. In this work, we propose to embed self-similarity as an architectural inductive bias to better resolve fine scales. We formalize this inductive bias as a recurrence with a shared operator across scales and prove when this operator can be parameterized efficiently with parameter count independent of the number of fine scales predicted, allowing it to be appended to existing models as a small module. Using this result, we introduce self-similar cascade decoders, a family of such modules that admit domain-specific realizations. Our Fourier basis instantiation, called the self-similar spectral cascade (S3C), improves performance on turbulent Navier-Stokes forecasting by up to 54% across backbones, resolutions, and time horizons—while contributing 0.1% of total parameters. Our wavelet instantiation, S2WC, outperforms several strong baselines for deterministic image super-resolution.