- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
Premise of the work is that visual information is distributed in redundant high dimensional population responses in the brain. The goal of the paper is to extract useful principles of how can this information be read out (by downstream circuits) through the lens of a transformer-based decoder trained to reconstruct the sensory input from recorded neural responses. Specifically, its attentional weights are used as a proxy for what a readout circuit might do with those responses during brain function. This is applied to data from monkey V1, in two conditions (clean and noisy gabor gratings) yielding some features that the authors consider realistic. Nonetheless, the very dynamic nature of the extracted readout makes this interpretation suspect from the perspective of biology.
Critical strengths
Interesting data in terms of effects of pixel level noise on V1 encoding at the population level
Critical weaknesses
A) The idea that the weights reflect a biological readout is to me fundamentally suspect, even if (as the authors advise) interpreted in terms of functional rather than synaptic connectivity.
The robustness of the representation coming from a distributed code means that potentially multiple readouts could get the job done. The argument here is that the self-attention can more directly take into account the coherence pattern of the population responses to decide what's a minimum useful subset of neurons that tell me what I need to know. I see at least two fundamental conceptual issues with this interpretation: 1) the computation of the self attention mask takes into account the full population, data that a downstream circuit is not expected to have. 2) related, but the readout being input specific is equally problematic since a downstream circuit cannot decide which neurons to read from until knowing what stimulus they are looking at, which fundamentally defeats the purpose of downstream information transmission. In other words, the downstream circuit cannot have enough knowledge to know which neurons to read from. As a separate issue, since there are multiple readouts that work, the degeneracy will likely be resolved via the inductive biases of the model, which are in no way set to reflect brain properties. It's not clear to me why any alignment between solutions would be more than happenstance.
B) Numerical comparisons to strawman models is superficial. Not clear how if any effort was put into making alternative nonlinear decoders work.
C) not clear why the results are deemed biologically sensible, or how much of the pattern preservation can be explained from the functionally-based way the neurons were indexed in the first place. Similarly -- redundancy is part of the premise so the fact that that is found as part of a more complex analysis seems like busywork.
D) found the figures generally unintuitive and difficult to see and interpret
E) other transformer based encoder-decoder solutions exist (e.g recent papers from Tolias lab), pixel level decoding has been done for retinal responses in a number of ways (e.g. Chichilnisky, Marre), the novelty of the approach is overstated
Questions for authors
What is the ground truth I, exactly? Regarding image reconstruction (as it appears on the screen?): Is the field of view large enough that the receptive field of the cells recorded span the full screen? how is that verified? are the gabors full field? What is the spatial resolution for the images to reconstruct?
What is the relationship between df/f in the data and the variable r?
Are dynamics over time something that the model is sensitive to?
How is the orientation selectivity of the cells determined (for their ordering in the model)?
Model controls: why have a nonlinear decoder of orientation as a label but not an MLP version of pixel level reconstruction?
The SSIM metric is not clearly providing a sense of the quality of the reconstruction, would it be possible to see some example reconstructed images for direct comparison? Also I can believe that a linear decoder out of the box does worse in terms of orientation estimation but why is the MLP version not any better? I find the control numbers counterintuitive. please explain
Missing definitions for variables, e.g. HxW is presumably the height and width of the image in pixels?
wouldn;t the alignment of high weights with neural tuning trivially be true even for a linear decoder with a weak sparsity constraint that picks different neurons across seeds? why is this finding interesting?
At which point in the analysis is the alternative linear decoder of I used for comparison? I seem to have missed that in the text
3 comments
Rebuttal received
I thank the authors for the clarifications but I still have a fundamental problem with the formalism and what it can possibly teach us about the nature of neural coding. I will maintain my score.
We thank the reviewer for their comments. We would like to clarify two aspects of our contribution.
First, regarding the computational formalism, our architecture was designed to mathematically characterize two key properties of neural populations: neuron-specific embeddings represent individual neuronal identities and functional heterogeneity, while self-attention captures response-dependent interactions among neuronal representations. Different population responses naturally produce different attention maps through neuron-specific embedding and self-attention, allowing the model to automatically assign neuronal weights without prior knowledge of stimulus identity or which neurons should be selected. Unlike conventional decoders with fixed (stimulus-independent) readout weights, our framework provides explicit stimulus-dependent attention maps, allowing us to examine how neuronal contributions and interactions vary across stimuli. This advances the interpretability of population readout beyond conventional decoding analyses. Previous computational studies have also explored potential biological implementations of Transformer-like computations through neuron–astrocyte networks (Kozachkov et al., 2023).
Second, regarding what our findings teach us about neural coding, our results highlight a computational principle of redundant encoding coupled with sparse effective readout, complementing classical efficient- and sparse-coding perspectives (Barlow, 1972). Building on the broader ideas of selective information gating and flexible routing in neural circuits (Yang et al., 2016; Wang & Yang, 2018), our study takes a further step by quantitatively characterizing stimulus-dependent selective readout using real V1 population responses and examining its flexibility and robustness through in silico neuronal removal and model retraining. Specifically, our results further show that distributed population information can support selective, stimulus-dependent readout, while redundancy allows functionally similar neurons to support alternative readout solutions following neuronal removal and model retraining, thereby preserving decoding performance.
Thus, our study advances both methodology and computational understanding of neural population readout. Methodologically, it enables interpretable analysis of stimulus-dependent neuronal weighting beyond decoding accuracy. Conceptually, it illustrates how redundant encoding can support sparse, flexible, and robust effective readout. Our findings provide a data-grounded, testable computational hypothesis for future causal investigation.
We sincerely thank the reviewer again for their thoughtful comments on both the conceptual framework and methodological details, particularly the experimental procedures, which have helped us clarify our arguments and improve the rigor and presentation of our manuscript.
References
-
Barlow, H. B. (1972). Single units and sensation: A neuron doctrine for perceptual psychology? Perception, 1(4), 371–394. https://doi.org/10.1068/p010371
-
Kozachkov, L., Kastanenka, K. V., & Krotov, D. (2023). Building transformers from neurons and astrocytes. Proceedings of the National Academy of Sciences, 120(34), e2219150120. https://doi.org/10.1073/pnas.2219150120
-
Yang, G. R., Murray, J. D., & Wang, X.-J. (2016). A dendritic disinhibitory circuit mechanism for pathway-specific gating. Nature Communications, 7, 12815. https://doi.org/10.1038/ncomms12815
-
Wang, X.-J., & Yang, G. R. (2018). A disinhibitory circuit motif and flexible information routing in the brain. Current Opinion in Neurobiology, 49, 75–83. https://doi.org/10.1016/j.conb.2018.01.002
Concern
A: We thank the reviewer for these important points.
(1–2) Access to the full population. Attention weights are computed from the population response itself, so no knowledge of stimulus identity is required. However, we agree that this computation requires access to the full population, which is unavailable to downstream circuits. We therefore do not propose attention as a biological mechanism. "Sparsity" refers to the effective readout weighting. Our claim is limited to showing that, under redundant coding, a selective readout exists that is efficient and stable. We now state this explicitly in the Limitations.
(3) Degeneracy and inductive biases. We agree that many readouts achieve similar accuracy and that the model's inductive biases select among them. Indeed, shared embeddings reach comparable accuracy without tuning alignment (Fig. 3A), so accuracy alone does not imply biological correspondence. The alignment we report, however, is unlikely to be happenstance. It is measured against independently estimated tuning never provided to the model, persists after shuffling neuron order (Fig. 3D), and is consistent across FOVs.
We interpret it as a property of the solution favored by this architecture, not a uniquely biological one, and have revised the Discussion to frame these results as a testable hypothesis about selective readout requiring causal, circuit-level validation.
B: We selected established linear, MLP, and no-attention baselines, clarified their rationale (Section 3.4), and detailed optimization (Appendix A.3).
C: Functional tuning is known properties of V1. Our goal is not to rediscover them, but to ask how a readout can selectively use redundantly coded information while remaining efficient and stable. Biological sensibility. We assess it through stimulus–tuning alignment: the model's selected neurons match independently measured orientation tuning never given to the model. This is consistent with causal evidence that functionally defined ensembles can drive perception (Carrillo-Reid et al., 2019; Marshel et al., 2019), but we do not claim it establishes a biological mechanism. Neuron indexing. To test whether the pattern follows from the functional ordering, we randomly shuffled neuron positions and retrained the model. Stimulus–tuning alignment and sparse selection were largely preserved (Fig. 3D), so are not imposed by the original ordering. Redundancy. Redundancy is a premise, not a finding. What we add is how the readout behaves under it: after removing the highest-weighted neurons and retraining, functionally similar neurons took their place and decoding accuracy was maintained (Fig. 3B–C). This shows that the sparse selective readout is stable under redundant coding.
We frame these results as a testable computational account, not an established biological mechanism.
E: Our main contribution is using stimulus-dependent attention maps, which conventional decoders do not explicitly provide, to investigate computational strategies underlying neural population readout, rather than proposing a novel Transformer architecture or image-reconstruction method.
Q1: The target was a 79 × 79-pixel (~1°) Gabor within each FOV's mapped pRF, not full-screen (Appendix A.1, Fig. A.2A).
Q2: r∈RN is the vector of ΔF/F responses of all N neurons (Sec. 2.2, lines 133–138).
Q3: The model does not explicitly capture temporal dynamics. Fluorescence signals were averaged over the baseline (−500 to 0 ms) and response (500–1,000 ms) windows to calculate a single ΔF/F value per neuron per trial.
Q4: Neurons were ordered by preferred orientation estimated from training-only Gaussian fits, and this ordering remained fixed across all conditions. Details are provided in Section 3.2 and Appendix A.2.
Q5: In fact, the no-attention baseline included in our study is an MLP-based image reconstruction model. We have clarified this baseline in the revised Section 3.4 (Baselines) and provided further architectural details in the Appendix A.3 (lines 789-795).
Q6: Example stimuli and reconstructions are now shown in Fig. A.2B. Although MLPs have greater nonlinear capacity, this need not improve generalization, likely because our limited dataset cannot exploit it. To further investigate this, MLPs with 1–5 hidden layers gave median orientation errors of 2.922°, 3.581°, 3.807°, 3.162°, and 4.475°, versus 2.83° for the linear decoder, so added depth did not improve decoding on our dataset.
Q7: Yes, H=W=79.
Q8: Linear decoder weights showed no clear systematic relationship with orientation preference (Appendix B.5). Stimulus–tuning alignment is a criterion for biological relevance, not a novel finding itself (see response to L92v, Concern 2).
Q9: The alternative image-reconstruction decoder is No-attention baseline. We have clarified this in Section 3.4 and provided implementation details in Appendix A.3.