acceptodds
Under review as a conference paper at ICLR 2027

Improving geometric consistency of neuronal functional embeddings

Abstract

Deep neural networks provide a powerful framework for learning functional embeddings of neurons from their responses to visual stimuli. These embeddings can reveal structure in neuronal function, but biological interpretations require their structure to be reproducible across model fits. We identify three sources of inconsistency in data-driven core-readout predictive models for mouse visual cortex. First, the functional basis of the embeddings spanned by the core is not orthonormal, leading to distorted Euclidean distances when computed naively. Second, learnt receptive-field locations vary across model fits. Third, the image encoders (cores) learn different functional subspaces across different model initializations. Here we show that the first issue can be resolved by whitening the functional basis. Whitening corrects the geometric distortion, improving the consistency of local neighborhoods and global structure across models trained with different random weight initializations. The other two issues are mitigated by introducing a new Retinotopic Factorized Linear readout (ReFL). ReFL improves the consistency of readout locations and makes core functional decomposition more similar. Combining whitening with the ReFL readout yields the most consistent functional embeddings. These results highlight the importance of both the orthonormality of the functional basis and the readout mechanism for the reproducibility of learned neuronal embeddings.

Primary area:
applications to neuroscience & cognitive science
Keywords:
representation learning, neuron embedding, deep neural predictive (encoding) models, mouse V1

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CAv3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper discusses the issue of inconsistency across different random seeds when learning functional embeddings of neurons in the mouse visual cortex using deep predictive models. The authors employ a shared CNN to extract visual features, using the readout weight vector for each neuron as its functional embedding. The authors identify three sources of this inconsistency: the non-orthogonality of the feature basis formed by the core channels, which distorts Euclidean distances, the drift in learned receptive field locations across different seeds, and the emergence of distinct feature decompositions within the core across different seeds. To solve this, the authors propose estimating the channel covariance post-training and applying a whitening transformation, while simultaneously performing an inverse transformation on the readout vectors to ensure that the model's predictions remain unchanged. To address the latter two issues, the authors introduce ReFL readout, which generates spatial masks directly from a neuron's cortical coordinates via a shared linear mapping.

Critical strengths

There are many studies use neuron embeddings from various models to discuss cell types and functional organization. However, few have systematically examined whether these embeddings remain stable across different random seeds. In this paper, the authors provide a comprehensive evaluation of this issue, assessing both global and local consistency.

Critical weaknesses
  1. I feel there is a flaw in the whitening approach: the embedding distance after whitening is, in essence, equivalent to comparing the similarity between the predictive responses of two neurons to a set of images, which is expected that this approach would prove more stable across different random seeds since it's primarily determined by the data itself. Given this, a more straightforward approach would be to use the neurons' responses to a collection of images whether derived from model predictions or from experimental recordings, directly as their embeddings. The paper doesn't compare its method against this direct baseline. So, it remains unclear whether whitening actually provides any added value.

  2. This paper's analysis focuses largely on the consistency of embeddings across different random seeds. However, it doesn't examine whether greater consistency leads to better alignment with known cell types, morphologies, or transcriptomic labels. A model could be consistently wrong, which I think is an oversight, given that the core motivation of this paper is to provide a reliable foundation for biological conclusions.

Questions for authors

Could part of the functional differences between neurons be encoded in the shape and size of the ReFL masks rather than in the readout vectors?

1 comment
AuthorsOpenReview ↗

Consistency is a prerequisite for biological validity and we added the response clustering baseline

Thank you for your thoughtful review and recognizing the importance of evaluating consistency across random seeds for biological reliability.

Below, we address the concerns:

W1 Embeddings indeed provide a data-driven way to compare neuronal functions, just like raw responses. Our contribution is not using embeddings for this purpose, as prior work has done so (Turishcheva et al.,2024; Nellen et al.,2025). Our contribution is recognizing and correcting for the non-orthonormal basis where these embedding lie – Fig. 1 illustrates why this is important. In fact, whitening corrects the flawed distance computations used in prior work and does not require additional data.

As for comparing embeddings with responses, we disagree that it is a direct baseline.

First, raw responses are dominated by stimulus drive (if there was any information in the receptive field) rather than real functional properties. For recorded responses, this means we can only compare neurons that see the same part of the image (have the same receptive fields), although neurons can have similar functions despite different receptive fields. The model separates receptive field position from function, allowing comparisons across all neurons. This is completely independent of whitening. Also, we lack ground-truth receptive fields and can only estimate them using the model.
Second, comparing ground-truth responses requires repeated presentations of the same stimuli sets, to cluster a set of presentations and measure consistency across repeats. However, raw responses do not account for pupil movements, which can change the input received by individual neurons even for the same stimulus.
Third, the response vectors (predicted or recorded) can be extremely high-dimensional - equal to the number of images, thousands for our train set. Computing distances in such high dimensions is problematic (Aggarwal, Hinneburg, Keim 2001). Functional embeddings compress neuronal responses, avoiding extremely high dimensions.

Despite these limitations, we performed the analysis using 9 repeats of 100 test images for 5 mice (2 lacked test sets; test images are shared across mice) and 2,553 neurons with receptive fields within 1 pixel of the population median. For each repeat (100 unique images), we clustered neuronal responses and model predictions and computed consistency metrics across repeats. Model predictions were more consistent, possibly because models account for pupil shifts, pupil dilation, and running speed, which modulate neuronal response amplitudes (Niell & Stryker, 2010).

To compare response and embedding consistency, we predicted 5,000 responses to unique training images from each model seed, clustered responses per seed and measured consistency across seeds. We used the same mice and neurons as before. Embeddings outperform predicted responses on all metrics (ARI, CKA, RSA, kNN), with ReFL outperforming Gaussian.

The plots are in the appendix W.

W2 We agree that high consistency does not guarantee biological validity. However, reproducibility across model initializations is a prerequisite for reliable biological interpretation. For example, deriving cell types from representations that cluster differently across random seeds would be unreliable. Our approach helps ensure that geometric analyses reflect stable representations rather than optimization stochasticity. We do not claim that consistency guarantees correctness. Rather, our claim is that biological conclusions drawn from models require both state-of-the-art predictive performance and stable representations, the former ensures that the model is not “consistently wrong”.

To show the link between consistency and biological validity, we analyzed embeddings before and after whitening on the marmoset retina dataset used in Nellen et al.(2025), with 4 ground-truth cell types. The model was trained on 235 cells, and consistency was evaluated on 145 labeled cells. Whitening improves all consistency metrics. Most notably, whitened embeddings show near-perfect ARI up to 4 clusters followed by a sharp drop at 5, while non-whitened embeddings show a smooth decay with much lower ARIs. Confusion matrices further support this result. Since the default model uses data-initialized receptive fields, adding retinotopy is not meaningful, so we cannot test ReFL on this dataset.

The plots are in the appendix V.

Q1 That's an interesting question. The mask could give some type of pooling across bigger receptive fields. We hypothesize that this may benefit cells exhibiting surround suppression (where a cell response is reduced when stimuli extends beyond the classical receptive field, such cells require integration over larger input size; Allman et al., 1985; Vaiceliunaite et al., 2013), but this hypothesis requires a separate study and currently unavailable experimental data.

We hope our reply addresses your concerns and helps to reconsider your assessment.

Reviewer_A89KOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes a new method called ReFL to improve functional embedding consistency globally and locally over the Gaussian readout. ReFL’s specific contribution is to predict an entire normalized spatial mask from cortical coordinates using a shared affine map, spatial smoothing, and softmax. The method mechanism is straightforward and easy to understand. The effectiveness is supported by superiority of ARI, linear KCA, RSA and kNN consistency.

Critical strengths
  1. The idea of ReFL is simple but effective. It provides a clear spatial inductive bias through a shared cortical-coordinate-to-mask mapping. The design the elegant and easy to implement.

  2. The experiment controls are substantial. Multiple consistency metrics, smoothing sweeps, retinotopy ablations, and frozen-core experiments support the observed improvements.

  3. The paper is good written and the presentation is easy to follow. Figure 3 illustrates the architecture of ReFL well.

Critical weaknesses
  1. The novelty is incremental, All components are well known. Whitening, factorized readouts, and retinotopic conditioning are established ideas; novelty lies mainly in their combination.

  2. Consistency does not establish biological validity. No ground-truth experiment shows that the embeddings recover neuronal functional similarities more accurately.

  3. It's difficult to track the attribution of improvements specifically to ReFL. Different pretrained cores and changes in spatial pooling complicate the mechanistic isolation.

Questions for authors
  1. Could you explain how can you establish that higher consistency reflects better recovery of neuronal function, rather than stronger architectural constraints or functional information shifting from channel weights into spatial masks?

  2. Can both readouts be trained on the same frozen core to isolate the readout’s contribution? The current frozen-core experiment uses different pretrained cores for the two architectures. You could try one more ablation experiment.

  3. How does ReFL compare with existing consistency methods under the same whitening procedure, and do its benefits persist with another core architecture or dataset? This can be answered by using more baseline methods.

I will consider raise my score based on the answers to those questions.

4 comments
AuthorsOpenReview ↗

Our contribution is correcting the distance computation, evaluating and improving consistency across scales. Consistency is a prerequisite for biological validity - we show this on a dataset with the known ground truth cell types

Thank you for your detailed feedback, acknowledging the effectiveness of our method and thoroughful controls for the experiments.

As for the weaknesses and questions:

W1 - Novelty The main contribution of our work is not in introducing new individual components, but identifying previously overlooked sources of inconsistency in both core and readout of predictive models and showing how a combination of several seemingly small components can mitigate these inconsistencies:

First, ensuring the embedding space to be orthonormal improves model and embedding consistency. This is not only relevant from a consistency point-of-view but also because non-orthonormal functional bases distort distances in embedding spaces (see Fig. 1). Several prior works have not accounted for this fact (Turishcheva et al., 2024; Nellen et al., 2025) but any downstream analysis that depends on distances between embeddings (e.g. clustering, similarity calculation etc.) will be affected.

Second, prior work assessed consistency primarily through consistency of clustering only, while we evaluate both local and global consistencies with various metrics and show improvements across scales. Local vs global distinction matters: consistent local neighborhoods alongside inconsistent clustering may indicate an absence of well-defined clusters, rather than variability across seeds. We will make it more clear in the paper.

Finally, despite the individual components being known - their combination is effective and ReFL shows a clear improvement in all consistency metrics. Moreover, we are the first one to analyze not only the embeddings consistency but also to look at the core functional subspaces these embeddings lie in, explaining why ReFL is more consistent than Gaussian readout. Fig. 7 shows that ReFL leads to cores learning a more similar functional basis compared to the Gaussian readout.

W2 - Biological validity We agree that high consistency does not guarantee biological validity. However, reproducibility across model initializations is a prerequisite for reliable biological interpretation. For example, deriving cell types from representations that cluster differently across random seeds would be unreliable. Our approach helps ensure that geometric analyses reflect stable representations rather than optimization stochasticity. Please see Q1 for further discussion.

Additionally, we analyzed consistency before and after whitening on the marmoset retina dataset used in Nellen et al., 2025. The dataset has 4 ground-truth cell types; the model is trained on 235 cells, consistency analysis is done on 145 labeled cells. Whitening improves all consistency metrics, but ARI highlights its connection to biological validity: whitened embeddings have near-perfect consistency up to 4 clusters (the number of ground truth cell types in that dataset), then a sharp drop at 5, while non-whitened embedding have a smooth decay with much lower ARIs. This shows whitening enables more reliable recovery of the underlying data structure, which is further supported by confusion matrices for a fixed seed. Finally, because default model receptive fields are initialized with the positions estimated from the experiment with white noise stimuli, adding retinotopy is not meaningful, so we cannot test ReFL here. For the plots - please see the appendix V in the updated pdf.

W3 - ReFL impact We believe that the improvement from ReFL is very straightforward to separate: Fig. 4D shows a clear gain in kNN consistency of whitened ReFL over whitened Gaussian, isolating the contribution of ReFL while accounting for the non-orthonormal functional basis, same for Fig. 4B-C. We prioritize whitened versions since distances are computed correctly there, though the non-whitened comparisons in Fig. 4B–D show the same trend. The partial explanation for the improved consistency is in Fig. 7, showing that ReFL also leads to more similar core functions learnt across initializations. The core architecture is identical across all experiments, all models in our main experiments are trained from scratch, with cores and readouts trained jointly. Frozen cores are used only in appendix control experiments. For more frozen core experiments see Q2.

AuthorsOpenReview ↗

Consistency plus performance enables reliable interpretation; frozen-core ablation; baselines

Q1 - Higher consistency vs neuronal function We have not claimed that higher consistency directly reflects better recovery of neuronal function. Rather, our claim is that biological conclusions should be drawn from models that have both state-of-the-art predictive performance and stable representations. ReFL is more reliable because it leads to learning more similar core functional decompositions across seeds (Fig. 7). We think stronger architectural constraints are not a problem if the predictive performance is maintained, as in our experiments. As for the redistribution of functional information from channel weights to spatial masks - we cannot currently quantify it and explicitly acknowledge this in the limitations. We hypothesize that this may benefit cells exhibiting surround suppression (where a cell response is reduced when stimuli extend beyond the classical receptive field, so such cells require integration over larger input size; Allman et al., 1985; Vaiceliunaite et al., 2013), but this hypothesis requires a separate study and currently unavailable experimental data.

Q2 - Frozen core To address your request to compare readouts on the same core, we extended the experiments from Appendix L (see updated pdf). Since jointly training a core and readout may favor fine-tuning the same readout type, we tested both Gaussian and ReFL cores. On the Gaussian core, both readouts perform similarly, with ReFL showing slightly more stable readout locations. On the ReFL core, Gaussian performance is slightly worse, while readout location distance pixel consistency remains similar. This may be because ReFL can learn smaller masks when needed, whereas Gaussian readouts cannot expand their receptive fields. However, the similar performance suggests this is not an issue for most cells. Overall, ReFL learns slightly more stable readout locations even on the Gaussian core, although the improvement is minor given the already high baseline.

Q3 - More baselines The consistency baseline in our experiments is the adaptive L1-regularized Gaussian readout (Turishcheva et al., 2024). Another approach is DECEMber (Nellen et al., 2025), which performs clustering during training. However, this requires whitening during training, whereas our current whitening approach is post-hoc (as acknowledged in the limitations), so the two cannot yet be combined. The approach of Akhmetali et al. (2026) substantially reduces predictive performance on mouse V1 (Fig. 6 in their paper), so we do not consider it a suitable baseline. We are not aware of other directly comparable consistency methods and would welcome specific suggestions.

We hope these comments shed some light on our motivation, technical and design choices. We also hope that it would help you to reconsider some criticism.

AuthorsOpenReview ↗

Discussion invitation

Dear Reviewer,

Thank you again for your feedback. We would be very happy to hear whether our response addresses your concerns, or if any further clarification would be helpful.

I thank the authors detailed responses, which resolve my major concerns about novelty and more baselines. I will raise my score to weak acceptance.

Reviewer_WVgBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper characterizes a core-readout framework designed to predict neural representations from visual stimuli whose unique operational pipeline seeks to mitigate three identified sources of inconsistency across model initializations: a non-orthonormal functional basis, variability in learned readout locations, and differences in the functional subspaces learned by the convolutional neural network core. This pipeline involves post-hoc whitening of the functional basis, which corrects distortions introduced when computing Euclidean distances between readout vectors, and introducing a Retinotopic Factorized Linear (ReFL) readout contrasted against the conventional Gaussian readout's spatial sampling mechanism with learned spatially smoothed masks conditioned on cortical coordinates via retinotopic mapping for more biologically constrained receptive field locations. Optimization coupled with this ReFL readout in turn leads to functional decompositions that vary less across seeds. Both whitening and ReFL demonstrate improvements in embedding consistency across independent model fits.

Critical strengths

The paper should be accepted considering its role in addressing the broader issue of functional embedding reproducibility by deep neural networks, which is essential to rectify for subsequent analyses by these networks to make accurate and interpretable functional claims about cell types in brain areas like the visual cortex. The introduction to ReFL gives a novel architectural contribution while whitening, despite being a standard preprocessing technique, is provided a strong motivation to treat the geometric distortions resulting from the non-orthonormal functional basis. A systematic comparison across various global stability and local neighborhood metrics (ARI, RSA, CKA, chance-adjusted mutual KNN consistency) is especially valuable. There is also the interpretation of cortical functional organization provided alongside Fig. 8, which should be more directly connected to Appendix E/Fig. 9 finding of weak correlation between cortical and functional embedding distances with no Gaussian/ReFL difference. Therefore, the authors may benefit from expounding on the benefit of ReFL being not the spatial organization provided across the cortical sheet but rather stabilizing the internal functional representations at the individual neuron level and corroborating that with the known “salt-and-pepper” organization.

Critical weaknesses

Main areas that would benefit from a more careful framing is the suggested readout-location variability as a source of inconsistency. While ReFL learning a more fixed set of receptive fields for the same neurons across initializations was initially propped as a potential explanation for more consistent functional embeddings by ReFL (Fig. 5), we see the shift-corrected readout locations are similar for both Gaussian and ReFL readouts (Fig. 6E). So it might be more prudent to prescribe the retinotopic mapping component, included in both readout types, as the mitigating factor in this form of inconsistency rather than a unique feature of ReFL.

Questions for authors

Only other outstanding questions to address involve adding a pairwise comparison test (i.e. Wilcoxon signed-rank test or permutation test for paired evaluations across the same model initializations and seeds) to establish statistical significance of whitening and ReFL contribution to global and local embedding consistency (Fig. 4) and model prediction correlation (Fig. 7B). Metrics like RSA and CKA even for Gaussian already appear compressed near upper bound of scale, and while tight error bars provide visual reassurance that model initializations are stable, hypothesis testing will reinforce strong methodological claims about geometric consistency rather than relying on qualitative inspection of bar heights.

Minor suggestions not influencing recommendation:

  • Introduce “image encoders (cores)” first time cores are individually referenced in previous sentence of abstract.
  • First sentence in the “Limitations and Future Work” Section is grammatically difficult to comprehend to the point of obscuring the rationale behind post-hoc whitening vs during training
  • Orthonormal basis label in “Ignoring the coordinate basis distorts local geometry” sub-section Q >> C to align with Fig. 1.
  • “ReFL readout makes the core learn more similar functions” section mislabels Fig. 7 as Fig.8.
  • Mathematical formulation and regularization strength hyperparameter description for the adaptive L1 penalty applied to the readout vectors should be included in the methods and appendix. Same for k-means clustering before explaining how ARI and cluster consistencies are computed.
1 comment
AuthorsOpenReview ↗

Response the weakness and added Statistical Testing

Thank you for appreciating our paper, and highlighting the importance and the broader impact of our work as well as our especially valuable systematic comparison across various metrics.

As for the points that you brought:
Retinotopy is not unique for ReFL (critical weakness): You are correct that both Gaussian and ReFL readouts incorporate a retinotopic mapping component, as we highlighted in Fig 3. We have not claimed that ReFL is more consistent than Gaussian readout due to the retinotopy or that retinotopy is unique for ReFL. We first hypothesised if the more consistent readout locations are the reason for improved embeddings consistency. We rejected this hypothesis since the core-shifts compensate for the readout location shifts in the case of Gaussian readout, which is shown in Fig. 6E as you correctly pointed out.

The key is Fig. 7: ReFL improves consistency primarily because its core's learnt functional basis is more consistent across seeds. A more similar core functional basis leads to more consistent readout vectors as they lie in more similar subspaces. We speculate that the functional decomposition is more consistent due to more stable readout locations, since both core and readout are trained jointly.

Please let us know if this addresses your criticism or if you meant something else.

Statistical tests: We agree that statistical testing would improve the results of our work, hence, we trained 14 seeds for both ReFL and baseline models and added the permutation test.

Specifically, we trained 14 seeds for both Gaussian and ReFL models and split them into 7 disjoint pairs, such that the pairs are independent. For each pair we compute the consistency metrics (CKA, RSA, kNN consistency, ARI) and run a version of the permutation test - the exact one-sided sign-flip test. Specifically, for each metric, we compute n=7n=7 paired scores (baselinei,newi)(baseline_i, new_i) corresponding to the metric values for the baseline and new models, respectively and their differences di=newi−baselineid_i = new_i − baseline_i. Under the null hypothesis that model type has no effect, the labels within each pair are interchangeable, therefore, swapping the labels preserves the magnitude of each difference while flipping its sign, meaning each difference is equally likely to be +∣di∣+|d_i| or −∣di∣-|d_i|. We therefore construct the null distribution by considering all possible combinations of sign flips and computing the mean difference for each. This gives the distribution of mean differences expected by chance. The p-value is the fraction of null mean differences that are at least as large as the observed mean difference.

We pick up a one-sided test because we want to show that ReFL is better than Gaussian and whitened is better than non-whitened, so the direction matters. The two sided test would only test if they are different. With n=7, the smallest possible p-value is 1/27≈0.00781/2^7 ≈ 0.0078.

When comparing Gaussian vs Gaussian whitened consistencies, we reach the smallest possible p-value for kNN consistency, CKA and RSA. For ARI p-value is 0.0078 for under 15 clusters included and starts growing afterwards, but it stays under 0.05 for under 40 clusters.

When comparing ReFL vs ReFL whitened, we again reach the smallest possible p-value for kNN consistency, CKA and RSA. But ARI goes above 0.05 for more than 10 clusters and keeps growing.

When comparing whitened Gaussian vs whitened ReFL: CKA p value ~ 0.0313, RSA p_value ~ 0.0156, kNN consistency stays 0.0078 and but for ARI the curves are basically the same under 30 clusters (p-value above 0.05).

For the plots - please see the appendix R in the updated pdf.

We will also improve the minor writing suggestions.

We thank you once again for constructive feedback and helping us to improve our paper. We are happy to answer any follow-up questions

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.