Toward a mechanistic understanding of inference in visual cortex and diffusion models

We describe a model of perceptual inference in primary visual cortex (V1) equivalent to a minimal diffusion model whose function can be readily understood from its parameters. We use an implicit recurrent network to define the denoising network and train it with implicit differentiation. The recurrent dynamics are derived by generalizing classical sparse coding’s inference dynamics through the addition of learnable pairwise interactions among latent units. After training on natural images, the model exhibits exceptionally good denoising performance, with an emergent ability to restore image features such as extended contours amid extreme visual ambiguity. The model nearly matches the behavior of standard, black-box diffusion architectures. Owing to the model’s simplicity, we provide a mechanistic understanding of how the model learned to resolve ambiguity. The model learns orientated receptive fields and a pairwise interaction matrix that links neurons with similar orientation tuning. These connectivity patterns mirror the structure of horizontal connections in the superficial layers of V1. Besides connectivity, our Jacobian analysis reveals a mechanistic explanation of how the recurrent circuit assigns high probability to a continuous family of natural structural deformations, including the deformation that allows the model to complete contours. Intriguingly, within this circuit, a large fraction of latent variables learn to disconnect from visual input altogether, essentially forming a hierarchical representation that appears to enforce global consistency among image features. Together, the model and results bridge two distinct domains: for neuroscience, they generate concrete, testable hypotheses regarding functional connectivity in recurrent neural circuits during perceptual inference tasks; for machine learning, they elucidate the internal mechanisms learned by diffusion models that allow them to generate infinitely many novel images from a finite training set.