- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces a non-factorial recurrent model trained according to denoising diffusion, based on the sparse coding model for latent representations in the visual cortex. The authors demonstrate how this architecture allows for closed form, analytic solutions to the gradients of the latent representations. These gradients reveal an interpretable decomposition of how basis functions in the sparse coding model activate and suppress one another, and the authors draw comparisons to observed contour completion in their model, an effect which is known to occur in the visual cortex. These interpretable dynamics involve increasingly higher-order multi-hop comparisons amongst active neurons' local orientations. Furthermore, they discover a subset of basis functions which have no direct mapping to pixel-space outputs, but play a crucial role in global dynamics, and suggest this may give insight into traditional UNet-based diffusion models. Finally, the authors make new predictions about visual cortex activity, specifically that in contour completion neurons will excite their neighbors strongest under noisy observations, and that the effects are context-dependent.
Critical strengths
- The paper applies clean mathematical treatment to their setup, and breaks the architecture down into specific mechanistic principles which are insightful into how sparse coding could result in meaningful latent representations of observed imagery under a diffusion objective. I feel that I have genuinely deeper insight into a connection between sparse coding and diffusion, and how sparse coding can model the visual cortex, after reading this paper.
- The authors draw meaningful comparisons across visual cortex findings and diffusion models using their new non-factorial sparse coding model, and include a convincing comparison to a parameter-matched UNet based diffusion model, demonstrating similar performance.
Critical weaknesses
- The mechanistic results seem quite specialized to this specific non-factorial sparse coding architecture. Since the architecture's assumptions are closest to the neuroscience setting, the value added to understanding of the brain seems to be the strongest. Although the authors acknowledge the difference, the main writing may still be over-emphasizing the level of insight this provides for diffusion broadly. E.g. it's not clear to me that the notion of basis functions and especially the global synchronization effects apply to a UNet or DiT.
- Relatedly, the final takeaway is not entirely clear. The alignment of generative diffusion objectives with visual cortex is quite interesting, but the final claims are fairly modest augmentations to current theory predictions of neural activity in V1.
- Less critically, the presentation could be improved in a few places. Specifically, earlier introduction to the notion of the basis function dictionary and a more intuitive outline of the sparse coding architecture implementation and how it differs or relates to standard diffusion would likely make the paper easier to read.
Questions for authors
- Please elaborate on how the principles of basis functions and global synchronization in the non-factorial sparse coding architecture may transfer to typical diffusion (UNet and/or DiT). In general, I'd be interested to hear a clarification of what findings you expect to transfer over, and which seem specific to your implementation.
- Please clarify what you see as the primary contributions of the research, and how it may advance our understanding or future design of generative diffusion models (if you see that as a possibility). This is related to question 1, as the paper does not seem to state how the mechanisms of generalization found in the sparse coding architecture would or would not translate to diffusion processes in general.