acceptodds
Under review as a conference paper at ICLR 2027

A pipeline for interpretable neural latent discovery

Abstract

Mechanistic understanding of the brain requires interpretability of large-scale neuronal computations. Many latent variable model approaches applied to such datasets excel at decoding but yield opaque latent spaces. We address this with NLDisco, a pipeline for interpretable neural latent discovery. Motivated by successful applications of sparse dictionary learning in AI mechanistic interpretability, NLDisco encourages hidden layer neurons in sparse encoder-decoder models to learn interpretable structure in dense neural activity. A flexible, open-source software package supports interpretable latent discovery across recording modalities and experimental paradigms. We validate the pipeline on both synthetic and real extracellular recordings across species and brain regions, showing that it recovers ground-truth features in synthetic data and reveals meaningful representations in real data. These results highlight the pipeline's potential to advance neuroscientific discovery and provide a foundation for future applications and extensions.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Neuroscience, Interpretability, Latent Variable Models, Representational Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YXrQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

NLDisco is an open-source pipeline for finding interpretable latents in neural population activity by training shallow, overcomplete, sparse encoder-decoder (SED) models, i.e., sparse dictionary / sparse autoencoder variants, with batch top-k sparsity so that individual latents act as candidate feature detectors. It introduces window SEDs (a flat-window encoder and a transformer-window encoder over SS timebins, with an optional shift-equivariant mode) and Matryoshka nested levels to handle multi-scale and absorbed features, plus an evaluation suite adapted from SAEBench. It is validated on simulated hippocampal place-cell spike data (ground-truth recovery), macaque motor-cortex data (comparison against CEBRA and LangevinFlow on two chosen features), and an 18-hour mouse RSC/SC foraging recording (one latent tracks wheel activity).

Critical strengths
  • The pipeline is complete, usable and well motivated. From spike-sorting ingestion through training, evaluation and latent inspection, NLDisco lowers the barrier to sparse-latent analysis of neural data and supports multiple modalities and paradigms, which is consequential for adoption by experimental neuroscientists.
  • The synthetic demonstration is a clean, falsifiable sanity check. Showing that matched latents recover simulated place fields, and that Matryoshka levels recover nested broad/narrow fields with source-activity traceback, convincingly demonstrates that the sparse dictionary behaves as intended.
  • The architectural adaptations are thoughtful. Window SEDs and Matryoshka levels are sensible, low-overhead responses to the temporal and multi-scale structure of neural data, and the shift-equivariant mode is a natural extension.
Critical weaknesses
  • The evaluation does not support the central interpretability claim. Latent interpretation is manual and subjective, feature selection is post hoc, and the motor-cortex comparison is explicitly a targeted search for pre-specified features. Cov/Spec/Sel/AUROC are computed on latents chosen to match known features. Without a quantitative, selection-bias-controlled measure of interpretability or discovery yield, the paper cannot substantiate that NLDisco yields more interpretable latents than CEBRA or LangevinFlow, i.e., this might not be guaranteed.
  • Methodological novelty is limited and benchmarking is narrow. The pipeline largely re-packages established sparse-dictionary techniques; comparisons use only two baselines and do not report standard reconstruction/decoding benchmarks such as NLB-style co-smoothing, so the contribution reads as primarily software rather than a demonstrated advance in neural representation learning.
  • Stability of the discovered features is not quantified. It is unclear how robust the reported latents are across random seeds, dictionary widths, sparsity levels, and sessions or animals, which matters because the scientific claims rest on individual hand-inspected latents.
Questions for authors
  • Can you provide a quantitative, selection-bias-controlled measure of interpretability or discovery, for example, automated feature-labeling agreement, inter-rater reliability from a human study, or discovery recall over a held-out feature set?
  • How stable are the reported features across random seeds, dictionary widths, sparsity levels, and across sessions or animals for the real datasets? Reporting stability statistics seems necessary.
  • Since the only data requirement is a predefined spatiotemporal binning, how does NLDisco perform on standardized reconstruction (is the reconstruction as good as baselines) and decoding benchmarks?
2 comments

Thanks for the constructive review! We’ll attempt to consolidate your critiques and questions into individual points we’ll respond to one-by-one, and answer each in-line in our reply. We believe we cover all critiques and questions brought up in your review, but if we left something meaningful out please let us know!

Our full response covers more than 5,000 characters, so we’ll leave this over multiple replies.

We will also ask you for clarifications in bold.


  1. “The evaluation does not support the central interpretability claim. Latent interpretation is manual and subjective, feature selection is post hoc, and the motor-cortex comparison is explicitly a targeted search for pre-specified features. Cov/Spec/Sel/AUROC are computed on latents chosen to match known features. Without a quantitative, selection-bias-controlled measure of interpretability or discovery yield, the paper cannot substantiate that NLDisco yields more interpretable latents than CEBRA or LangevinFlow, i.e., this might not be guaranteed.”

Can you confirm what you mean exactly by “the evaluation does not support the central interpretability claim”?

We agree that our current experiments do not fully establish that NLDisco definitively yields more interpretable latents or more reliable end-to-end discovery than CEBRA-Time or LangevinFlow. To address this limitation, we are considering an additional end-to-end discovery comparison on a second, new Aeon recording (4.3), that would pursue two of your suggested analyses. First, we will define a set of behavioral features independently of all models, search for them using the same procedure across methods, and evaluate the selected latents on held-out data. This would let us quantify how often each method recovers the independently specified features. Second, we will ask multiple raters to independently assess interpretations of a sample of latents from each method. We would independently inspect latents from each method on a “training” / “discovery” split, record candidate feature descriptions, and evaluate the selected coordinates and descriptions on held-out data using shared criteria. We would report the number and proportion of inspected coordinates yielding validated interpretations, as well as any distinct features surfaced by each method. This analysis would require substantial effort, but we will investigate whether we can complete it before the rebuttal deadline.

We recognize that neither analysis alone provides a universal measure of interpretability. Together, however, they would help assess the consistency of interpretations, and we believe they would satisfy your ask for a “selection-bias-controlled measure of interpretability” – we would be interested to hear if you agree, and why if not?

We would also like to address these points: 1a. “Latent interpretation is manual and subjective” 1b. “the motor-cortex comparison is explicitly a targeted search for pre-specified features” 1c. “feature selection is post hoc” 1d. “Cov/Spec/Sel/AUROC are computed on latents chosen to match known features.” On 1a: We agree that interpreting real-data latents involves manual inspection and judgment. In our experiments, we used visualization dashboards to examine latent activations alongside neural activity and behavioral variables, with AI agents assisting in the analysis of candidate features. We will expand the appendix to describe this workflow and how we selected latents for inspection.

On 1b: To clarify our results in 4.2, the behavioral descriptions were developed by inspecting a reference NLDisco model, then used to search for corresponding features in a separately trained NLDisco model and the baseline representations: we did not search for pre-specified features.

On 1c and 1d: we’re a bit confused because it seems 1c contradicts 1b and 1d: claims of the feature being “pre-specified” vs. “post hoc”. To be clear, we believe the reported metrics in 1d provide a useful comparison of how individual latents align with these descriptions. Features aren't known a priori: we see when a latent fires, then try to come up with a description of that occurrence (i.e. a feature), and then see how well the latent activity matches the features, scored by these metrics.

  1. “Methodological novelty is limited and benchmarking is narrow. The pipeline largely re-packages established sparse-dictionary techniques; comparisons use only two baselines and do not report standard reconstruction/decoding benchmarks such as NLB-style co-smoothing, so the contribution reads as primarily software rather than a demonstrated advance in neural representation learning.”

We agree that much of NLDisco brings together established sparse-dictionary techniques. However, we see this as a strength: practicing researchers benefit from a coherent pipeline with sensible defaults that can also be customized to their needs. Combining and adapting existing ideas is a common and productive approach in modern ML; for example, the Transformer architecture (Vaswani et al. 2017) brought together existing deep learning mechanisms, e.g. self-attention and skip connections.

In addition, the paper introduces the Window SED, which extends sparse encoder-decoders to capture neural representations over temporal windows, and adapts Matryoshka nesting to address representations at different scales.

Regarding baselines, in addition to CEBRA-Time and LangevinFlow, we compare NLDisco to PCA in the synthetic experiment (Section 4.1).

Regarding reconstruction and decoding benchmarks: our primary goal is to discover interpretable neural representations, and we include decoding as a secondary evaluation to show that NLDisco’s representations retain useful behavioral information alongside their interpretability. NLB-style co-smoothing directly measures the predictability of held-out neural activity, but we did not include it because our primary focus is interpretability.

Given these goals, does the rationale for our evaluation choices make sense? Are there particular baseline methods or analyses you would still like to see, and what question would they help address?


  1. “Stability of the discovered features is not quantified. How stable are the reported features across random seeds, dictionary widths, sparsity levels, and across sessions or animals for the real datasets? Reporting stability statistics seems necessary.”

We agree that a systematic analysis of feature stability across training configurations and recordings would strengthen the paper.

We do show some evidence of recurrence in the current experiments. In Section 4.1, the synthetic place-field features are recovered across five independently trained models with random seeds, and we report the mean and variability of their activation curves. In Section 4.2, behavioral descriptions developed from a reference model with dictionary width 256 can also be matched to latents in a separately trained model with width 192.

However, we agree that these examples do not provide a systematic characterization of real-data feature stability across seeds, dictionary widths, or sparsity levels. To address this, we will investigate reporting recovery rates and variation in feature-association metrics across multiple seeds and training configurations. For the proposed statistics, a straightforward choice is the fraction of runs in which each feature is recovered. We also plan to analyze a second, new Aeon dataset (4.3) to assess whether similar features can be found in another session and animal. We would welcome your feedback on whether these analyses would address your concern.


  1. Since the only data requirement is a predefined spatiotemporal binning, how does NLDisco perform on standardized reconstruction (is the reconstruction as good as baselines) and decoding benchmarks?

As mentioned in our response to point 2, our primary focus is interpretable feature discovery. We evaluate reconstruction quality within NLDisco and report behavioral decoding comparisons with CEBRA-Time and LangevinFlow in the macaque experiment, but we do not currently report a standardized reconstruction or decoding benchmark. We agree, however, that scoring on a reconstruction benchmark would be be useful complementary evidence about the information retained by NLDisco’s representations, so we will aim to include this before the rebuttal deadline.


We hope these clarifications and suggested edits are clear, and we look forward to your response, and to any additional specific suggestions you may have. Thanks again for the thoughtful review!

Reviewer_8ZX3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents NLDisco, a pipeline for finding interpretable latent variables in neural population recordings using sparse encoder-decoder models. The main idea is to make individual latent units themselves the object of interpretation, rather than first learning a dense latent space and then searching for meaningful directions inside it. The paper also introduces temporal variants that operate on windows of neural activity, allowing a latent to correspond to a spatiotemporal pattern rather than a single time bin. The paper evaluates the approach in three settings: synthetic hippocampal data with known place-cell structure, macaque motor-cortex recordings during reaching, and a naturalistic mouse foraging dataset. On the synthetic data, the method recovers the intended spatial and temporal structure. On the macaque data, the authors compare selected NLDisco latents against CEBRA-Time and LangevinFlow on several behavioral features. On the mouse data, they show an example of a latent that is strongly associated with wheel-running behavior. The overall goal is not just decoding, but making the learned representation easier to inspect and use for hypothesis generation.I think the paper is addressing a real problem. A lot of neural latent-variable work is useful for prediction but leaves the scientist with a representation that is difficult to reason about. The sparse-latent framing is therefore sensible, and the software contribution could be useful in practice. My main hesitation is about what the experiments actually establish. The paper shows fairly convincingly that NLDisco can produce interesting and interpretable-looking latents. I am less convinced that the current experiments show that it is a better or more reliable method for discovering interpretable structure than existing representations.

Critical strengths

The strongest part of the paper is the formulation itself. Treating individual sparse latents as the basic unit of analysis is a reasonable way to make neural representations more inspectable, and the Window SED extension is simple enough that its scientific use is immediately understandable. The synthetic experiments are also useful. They show that the method can recover known place-field structure and temporally ordered features in a setting where the ground truth is actually known. That gives the method a meaningful sanity check rather than relying entirely on post-hoc interpretation of real neural data. I also like that the authors do not oversell decoding performance. They explicitly separate “the information exists somewhere in the representation” from “there is an individual latent that corresponds cleanly to an interpretable feature,” which is exactly the distinction the paper should care about.

Critical weaknesses

The main weakness is the real-data comparison protocol. For the macaque experiment, the behavioral feature descriptions are first developed by inspecting a reference NLDisco model. The paper then fixes those descriptions and asks whether corresponding dimensions can be found in a newly trained NLDisco model, CEBRA-Time, and LangevinFlow. The authors are transparent about this and explicitly describe the experiment as a targeted search rather than a comparison of unconstrained discovery. That makes the comparison valid for one question: “if NLDisco suggests this feature, does the same information also appear in other representations?” But it does not answer the stronger question I care about for this paper: “which method is actually better at surfacing meaningful features that a scientist would discover from scratch?” This matters because another method might expose equally interesting structure that simply does not look like the particular features chosen from NLDisco. Those features would never enter the evaluation. So I do not think Table 1 should be read as strong evidence that NLDisco is generally better for interpretable discovery, even though it does show that some of its latents are cleaner feature detectors for the selected behaviors. The paper itself acknowledges that the comparison does not measure end-to-end discovery effort.
The second issue is that the real-data interpretability process remains quite manual. The workflow is essentially: inspect latent activations alongside behavior, formulate a possible interpretation, check counterexamples, and then quantify the association. That is a perfectly reasonable exploratory workflow, but it gives the analyst a lot of freedom. The mouse wheel-running example is interesting, but it is still a selected example from one recording, and the authors appropriately note that other correlated behaviors could explain the association.
For me, this is more important than whether the paper includes one more benchmark or several more random seeds. The main open question is not whether NLDisco can produce an interpretable latent—it clearly can—but how often it does so, and whether the process is reliably better than inspecting other representations.

Questions for authors

For the macaque experiment, do the authors agree that the current comparison is primarily a feature-recovery comparison conditioned on features initially identified with NLDisco, rather than a comparison of end-to-end feature discovery? If you have any experiment where the behavioral features were specified independently of NLDisco, I would find that much more convincing.

In the naturalistic mouse experiment, roughly how many latents were inspected before selecting the wheel-related example? More generally, what fraction of active latents could be assigned a reasonably stable and nontrivial interpretation? I am not asking for exhaustive annotation of the whole dictionary, but some sense of the denominator would help a lot.

For the dense baselines, activation thresholds are calibrated using the activation frequency of the reference NLDisco latent. How sensitive are the threshold-dependent metrics (coverage, specificity, selectivity) to this choice? AUROC is less affected, but the thresholded metrics seem somewhat asymmetric by construction.

5 comments

Thanks for the thoughtful review! We’ll attempt to break down your critiques and questions point-by-point, and answer them each in-line in our reply: we'll form these points either as direct quotes from you, or through slight rephrasing in our own words. We believe we cover all critiques and questions brought up in your review, but if we left something meaningful out please let us know!

Our full response covers more than 5,000 characters, so we’ll leave this over multiple replies.

We will also ask you for clarifications in bold.


  1. “I am [not] convinced that the current experiments show that [NLDisco] is a better or more reliable method for discovering interpretable structure than existing representations…another method might expose equally interesting structure that simply does not look like the particular features chosen from NLDisco…The main open question is not whether NLDisco can produce an interpretable latent—it clearly can—but how often it does so, and whether the process is reliably better than inspecting other representations”

We agree that our current experiments do not fully establish that NLDisco is better or more reliable than other representations for end-to-end feature discovery. As you point out, in 4.1 and 4.2 we do a comparison of what might better be called “feature recovery” than true “feature discovery”.

However, we claim that NLDisco is designed to make individual sparse latents, their activating examples, and their contributions to neural activity a direct starting point for interpretable feature discovery, a design principle that other methods lack. We could (and do) attempt feature discovery in other methods by e.g. looking at PC coefficients in PCA, embedding dimensions in CEBRA, and latent coordinates in LangevinFlow, but none of these latent representations are as amenable to feature discovery as NLDisco’s SED latents (we can simply inspect when hidden neurons are active).

To address your question more directly, we are considering an additional comparison on a second, new, Aeon dataset (4.3). We would independently inspect NLDisco latents, PCA components, CEBRA-Time embedding coordinates, and LangevinFlow latent coordinates on a “training” / “discovery” split of the dataset, and record candidate interpretations before evaluating them on held-out data. This would let us examine which meaningful features each method surfaces, including whether the other methods reveal features that we do not find in NLDisco. Such an analysis would require a substantial amount of work, but we will see whether we can complete it before the rebuttal deadline. We are interested to hear whether this comparison or something similar would address your concern.

Here we also want to address your statement: “I do not think Table 1 should be read as strong evidence that NLDisco is generally better for interpretable discovery” and say that we fully agree – this table is just meant to show that NLDisco is competitive with other SOTA Neural LVMs for feature decoding.


  1. Feature selection favors NLDisco: The macaque behavioral feature descriptions are developed by inspecting a reference NLDisco model and then used to search a new NLDisco model, CEBRA-Time, and LangevinFlow. This conditions the comparison on features initially surfaced by NLDisco.

Yes, we agree, as we mention above. We believe we make this limitation explicit in the paper, but please let us know if you disagree!

And we still think this targeted comparison was useful: it allowed us to show that these features can be recovered across separately trained NLDisco models, and compared in CEBRA-Time and LangevinFlow models, and allowed us to perform this comparison using a shared set of metrics.

As discussed above, we are also considering an independent discovery analysis on a second, new Aeon dataset, in which we would inspect representations from each method separately and evaluate the resulting candidate interpretations on held-out data. This would help address the selection asymmetry and allow each method to surface potentially different features.


  1. “Real-data interpretability process remains quite manual”

We agree that interpreting latents in datasets involves manual effort, and chalk this up as an inevitably of scientific discovery – for now – and understanding – forever (even if in the not too far future all scientific discovery becomes automated in some sense, we can’t outsource our understanding =p).

In our experiments, we used visualization dashboards to explore latent activations alongside neural activity and behavioral variables, with AI agents helping analyze candidate features. Researchers then evaluated the proposed interpretations using the available data.

We will expand the appendix to describe this workflow, including how latents were selected for inspection and how the dashboards and AI assistance were used.

Feature-inspection dashboards developed for sparse autoencoders applied to language models, such as those from Anthropic and Neuronpedia (https://transformer-circuits.pub/2023/monosemantic-features/vis/a1.html, https://docs.neuronpedia.org/sparse-autoencoders/feature-dashboards), provide related examples of such exploration.

We also agree that the time and effort involved is an important practical consideration; we have not yet quantified the time or effort required or compared this across methods, but will attempt to do so in the end-to-end feature discovery comparison we mention above.


  1. “The mouse wheel-running example is interesting, but it is still a selected example from one recording, and the authors appropriately note that other correlated behaviors could explain the association.”

We agree that the wheel-running latent is a selected example from a single recording, and that its association with wheel movement does not guarantee that wheel movement itself is uniquely represented: we do show that the association persists across bouts spanning hours and after controlling for animal position, though other correlated variables such as mouse posture remain possible explanations. In the appendix, we show additional latents associated with other measured aspects of behavior in this experiment, including movement speed and direction. More generally, we view NLDisco as a tool for generating interpretable hypotheses about neural representations. The strength of an interpretation depends on the variables measured and the controls available in the experiment. NLDisco gives a researcher interpretable features, and then the researcher must decide if controls are already in place to make strong claims about a feature (e.g. an SED latent is sufficiently selective for environmental stimuli X), or if there are other confounds the current experiment doesn’t control for that need to be considered before making a strong claim (e.g. environment stimuli X and Y always occurred together, so a future experiment must disentangle them).

  1. “For the macaque experiment, do the authors agree that the current comparison is primarily a feature-recovery comparison conditioned on features initially identified with NLDisco, rather than a comparison of end-to-end feature discovery? If you have any experiment where the behavioral features were specified independently of NLDisco, I would find that much more convincing.”

Yes, as mentioned above, we agree with this assessment that our results in 4.2 are primarily a feature-recovery comparison than an end-to-end feature discovery comparison, and we mention above the feature discovery comparison we will seek to complete before the rebuttal deadline.

Our synthetic experiment provides an example in which the features are specified independently of NLDisco through the data-generating process. There, individual NLDisco latents recover ground-truth place fields that PCA components do not cleanly separate. We acknowledge that this synthetic comparison does not replace an independently specified comparison on real neural data, but beyond such synthetic experiments, we believe it’s hard (impossible?) to know which behavioral or environmental features are represented in neural data.

This seems a bit hacky, but we could do something like prespecify behavioral conditions from the macaque task, such as reach direction or movement onset, independently of inspecting any model’s latents. We could then search for those conditions using the same procedure across methods and evaluate the selected latents on held-out trials. This is similar to what we’ve done, but instead of using a “seed” NLDisco run to discover features that we then replicate, here we could seed the features directly.

We are not sure if this would address your concern, so please let us know if we’re misunderstanding your point here, or if you have an example of an experiment that would address your concern!

  1. “In the naturalistic mouse experiment, roughly how many latents were inspected before selecting the wheel-related example? More generally, what fraction of active latents could be assigned a reasonably stable and nontrivial interpretation?”

Great question! We acknowledge that the inspection required and the fraction of latents that are interpretable are important for assessing utility for researchers who may want to use NLDisco. Models trained on the Aeon dataset in 4.3 used a 192-latent dictionary; we quickly inspected about half of these, and looked for latents that were sparsely active and correlated with behavioral variables we thought may be represented (e.g. position, speed, direction). Of these, we found about 10 candidates, and subjectively picked 4 that we deemed most interpretable (one shown in the main text and three in the appendix).

In our experience with this and similar datasets, we estimate that we typically find between 10-30% of latents yielding interpretable features, though we have not systematically recorded this and cannot say this with certainty. Additionally, this is of course highly dependent on the task performed and the brain regions recorded from (e.g. if an experiment involves a fine motor movement task performed by macaques with high spatiotemporal behavior recordings and neural data from primary and secondary motor cortex, we expect to find more interpretable features than we found in the Aeon dataset).

We will investigate whether we can add a systematic analysis in which we inspect a random sample of active latents, document the interpretation criteria, and evaluate candidate interpretations on held-out time periods.


  1. “For the dense baselines, activation thresholds are calibrated using the activation frequency of the reference NLDisco latent. How sensitive are the threshold-dependent metrics (coverage, specificity, selectivity) to this choice? AUROC is less affected, but the thresholded metrics seem somewhat asymmetric by construction.”

If by ”dense baselines” you mean CEBRA-Time and LangevinFlow, then we did not calculate this sensitivity. This is a good point to bring up, we agree that these metrics may be sensitive to the activation-rate matching procedure, and we will do this and add results to the Appendix.


We hope these clarifications and suggested edits are clear, and we look forward to your response, and to any additional specific suggestions you may have. Thanks again for the thoughtful review!

Reviewer_9Vz6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The manuscript proposes the use of sparse autoencoders for spiking neural activity to identify interpretable latent states given the behavior data. Across synthetic experiments, the authors show that their method can identify latent states matching the underlying firing rates, and across a real-world motor spiking dataset, they show that the proposed method again identifies latent states tuned for specific behavior conditions.

Critical strengths
  • The manuscript is clearly written, and the methodology and evaluation protocols are easy to follow.
  • The authors developed multiple versions of their model with different backbones, input representations, and SAE structures.
Critical weaknesses
  • I find the methodological contribution of the proposed method fairly limited. I could not find any methodologically novel component within the model formulation, and the input representations experimented with here are commonly used within the domain of neuroscience (binning the spikes).
  • The proposed solution seems to support single-session neural activity only. Given the recent advancements in foundation-style modeling of neural activity improving behavior decoding accuracy significantly, the proposed method seems to fall short in this aspect, and extending it to support this setting does not seem trivial for the proposed method. Further, I think interpretability is most valuable when it comes with generalizability, and the current version of the work does not seem to enable that.
  • In addition to the point above, the method's interpretability relies on the supervised behavior and manual search over latent states' activation trajectories within the session of interest. In general SAE literature, interpretations of the individual latent states are usually obtained in the training set, and their generalization to test data is demonstrated (Tinaz et al., 2025, https://arxiv.org/abs/2504.15473). The proposed method's single-session training mode and its reliance on behavior signals of the session of interest are significant limitations in this aspect.
  • Importantly, from a practical point of view, I am having a hard time finding a utility for the proposed approach. If we need the supervised behavior data to identify interpretations, why wouldn't someone train a supervised model of neural data and behavior directly? One potential advantage of the proposed solution could be extracting interpretations beyond behavior, which is not shown in the paper.
  • Can the authors explain how they identify the latent states whose activation probabilities align with the behavior/underlying rates? Do they do a manual search over the latent states?
  • Several important recent works on latent state space models of neural activity are missing in the related work, such as switching LDMs, PSID, DPAD, DFINE, POYO, NDT, NDT-2, pi-VAE, etc. The coverage of the related work should be improved.
  • How sensitive is batch top-k selection to the batch size used in the training?
  • The authors' statement regarding CEBRA is incorrect: CEBRA-time is not among the best-performing models in the Neural Latents Benchmark; CEBRA-behavior is. This may seem a trivial detail, but as it has been shown across several works at this point, they can differ significantly in terms of downstream performance.
  • Even though they explain what dashed lines represent in Fig. 3 in the caption, it should be put in the figure legend too.
Questions for authors

Please see weaknesses above.

7 comments

Thanks for the thoughtful review! Happily, we can address each of your 9 concerns point-by-point, either through analysis + paper edits, or clarification. Our full response covers more than 5,000 characters, so we’ll leave this over multiple replies.

We will also ask you for clarifications in bold.

First, we’ll clarify your paper summary; you state “the manuscript proposes the use of sparse autoencoders for spiking neural activity” – we note that while the examples shown in the paper all use an SAE on neural spiking data, we explicitly mention that the general form of the model used in NLDisco is an SED, and can be applied for multiple purposes on multiple types of neural data. We have collaborators already applying NLDisco to neural calcium fluorescence data from Miniscope calcium imaging in mice for cross-region prediction and “brain-diffing” between subjects, points we mention in “Future work”. While this work is out-of scope for this paper, we’d be happy to make this generality more clear in our current paper, and would be happy to hear suggestions on how to do so!


  1. I find the methodological contribution of the proposed method fairly limited. I could not find any methodologically novel component within the model formulation, and the input representations experimented with here are commonly used within the domain of neuroscience (binning the spikes).

We disagree with this statement. We show that NLDisco can recover ground-truth synthetic neural representations (4.1), find expected neural representations in a popular open-source dataset (4.2), and find novel neural representations in a recently published open-source dataset (4.3). Individually and together, these results show the method’s utility for practicing Computational and Systems Neuroscientists. Further, we note in the paper that we introduce the “novel yet conceptually simple Window SED (W-SED)”, which allows models a way to discover representations that emerge over varying timescales.

We agree that much of the pipeline brings together existing methods, but we see this as a strength. Practicing researchers benefit from a coherent pipeline with sensible defaults that can also be customized to their needs. Combining and adapting existing ideas is a common and productive approach in modern ML; for example, the Transformer architecture (Vaswani et al. 2017) brought together existing deep learning mechanisms, including self-attention and skip connections.


  1. The proposed solution seems to support single-session neural activity only. Given the recent advancements in foundation-style modeling of neural activity improving behavior decoding accuracy significantly, the proposed method seems to fall short in this aspect, and extending it to support this setting does not seem trivial for the proposed method. Further, I think interpretability is most valuable when it comes with generalizability, and the current version of the work does not seem to enable that.

For this point we offer some clarifications, and ask for some clarifications as well! NLDisco can be applied to multiple sessions, and the pipeline supports session-level splits and prevents training windows from crossing session boundaries. When the same neurons can be tracked across sessions, a model can be trained on multiple sessions and evaluated on held-out sessions.

Comparing representations across recordings with different neurons is a separate challenge. Models can be trained independently, but their features must then be compared using a common reference, such as behavioral or environmental variables. We have collaborators currently applying NLDisco in a cross-subject setting, as mentioned above.

We strongly agree with your statement here: “I think interpretability is most valuable when it comes with generalizability”, and would be happy to demonstrate the reproducibility of discovered representations across subjects – would this analysis address both the multi-session and generalizability concerns?

Overall, we think there were two main critiques in this point: (1) that NLDisco cannot be applied to multi-session data: This is not true, and we’d be happy to make this clearer in the paper. (2) that NLDisco does not enable showing generalizability of interpretability. This is also not true, but we agree that we don’t show this, and we should.

We could address both by adding to 4.3 similar results (wheel activity latents) from another mouse participating in the same experiment, would this address your concerns?

Please let us know if we’ve understood your critiques, and what sort of results would address them!

  1. In addition to the point above, the method's interpretability relies on the supervised behavior and manual search over latent states' activation trajectories within the session of interest. In general SAE literature, interpretations of the individual latent states are usually obtained in the training set, and their generalization to test data is demonstrated (Tinaz et al., 2025, https://arxiv.org/abs/2504.15473). The proposed method's single-session training mode and its reliance on behavior signals of the session of interest are significant limitations in this aspect.

Overall, we think we can break this point down into three critiques:

  1. The method's interpretability relies on the supervised behavior and manual search over latent states' activation trajectories

We clarify that training of NLDisco models is unsupervised with regards to behavioral or environmental variables: models only predict some target neural data from some source neural data, and when successful, one latent (model hidden layer neuron) learns a meaningful neural representation. Given this, researchers then of course need to understand what behavioral or environmental variable the latent corresponds to. This process, while manual in nature, can be made efficient through the use of visualization dashboards and assistance from AI agents. We are happy to elaborate on this in the paper if you feel we have not made this sufficiently clear?

  1. In general SAE literature, interpretations of the individual latent states are usually obtained in the training set, and their generalization to test data is demonstrated (Tinaz et al., 2025, https://arxiv.org/abs/2504.15473).

Yes, agreed! We show this as well in our results on datasets used in 4.2 and 4.3: in 4.2 all decoding metric values in Table 1 are reported on withheld data, and in 4.3 the metrics for the wheel activity latent (and other latents shown in the appendix) are also reported on withheld data.

We acknowledge that the SED itself was trained on neural activity from full session recordings – we could train on a subset train split of say 80%, find a natural variable that is represented by a latent in this train set, and then on the withheld 20% test set, find all the occurrences of this natural variable, and assess the latent’s activity for these occurrences – is this what you had in mind and would this address your concern?

  1. The proposed method's single-session training mode and its reliance on behavior signals of the session of interest are significant limitations in this aspect.

As mentioned above, NLDisco can be applied to multi-session data, and neuroscientists always need to inspect behavioral and environmental variables in order to understand what a neural representation encodes, regardless of whether or not they use NLDisco to expedite this process. If the concern is coverage-related, e.g. a latent learns a neural representation of an behavioral or environmental variable for which data is not collected, we agree this is a limitation that can exist in systems neuroscience experiments generally, and we advocate for rich data collection in such experiments, of both neural data, and behavioral + environmental data. Could you clarify if this is what you had in mind, or if we’ve misunderstood?


  1. Importantly, from a practical point of view, I am having a hard time finding a utility for the proposed approach. If we need the supervised behavior data to identify interpretations, why wouldn't someone train a supervised model of neural data and behavior directly? One potential advantage of the proposed solution could be extracting interpretations beyond behavior, which is not shown in the paper.

We believe the practical value of NLDisco is clear: discovering interpretable neural representations without behavioral or environmental supervision. We try to make this clear in the Abstract (and rephrased in the Introduction): “Many latent variable model approaches applied to [large-scale neural] datasets excel at decoding but yield opaque latent spaces. We address this with NLDisco, a pipeline for interpretable neural latent discovery.”

We agree that a supervised model is an appropriate and better choice when the goal is to predict a prespecified behavioral or environmental variable from neural activity. NLDisco addresses a complementary goal: exploring the structure of neural activity and discovering candidate representations without specifying behavioral or environmental targets during model training. Learned representations can therefore be examined without bias in relation to multiple variables, and novel variables a researcher may think of later, without needing to refit a model.

One potential advantage of the proposed solution could be extracting interpretations beyond behavior, which is not shown in the paper.

This is a great point, and we do have results showing that NLDisco latents correspond to environmental variables, such as properties of images / visual stimuli, and we would be happy to add this to the appendix. Is a result like this what you had in mind?


  1. Can the authors explain how they identify the latent states whose activation probabilities align with the behavior/underlying rates? Do they do a manual search over the latent states?

This is a great point, and we realize this was not made significantly clear in the paper. In short, yes, manual search over the latents was performed via bespoke experiment-specific dashboards (similar to Anthropic’s “feature browser” and Neuronpedia’s “feature dashboards” shown here: https://transformer-circuits.pub/2023/monosemantic-features/vis/a1.html, https://docs.neuronpedia.org/sparse-autoencoders/feature-dashboards), and expedited with AI agent assistance.

We will add examples of dashboards we used to the appendix, and also explain there how AI agents assisted with feature discovery.


  1. Several important recent works on latent state space models of neural activity are missing in the related work, such as switching LDMs, PSID, DPAD, DFINE, POYO, NDT, NDT-2, pi-VAE, etc. The coverage of the related work should be improved.

Thanks for bringing these works up! We definitely want to be sure we are referencing all relevant literature. Of note, we already cite DPAD in the Introduction, and ST-NDT and NDT-2 in Related Work and Positioning. We agree that we should also add discussion of switching LDMs, PSID, DFINE, pi-VAE, and POYO. In particular, we will clarify that several of these methods target interpretability, but focus on different objects (such as dynamical regimes, or latent manifolds), whereas NLDisco emphasizes individual, interpretable sparse features and their contributions to neural reconstruction. We will also discuss multi-session transfer as a distinct challenge addressed by foundation models such as POYO and NDT-2.


  1. How sensitive is batch top-k selection to the batch size used in the training?

Just to confirm, if we rephrase this slightly, we think you’re asking: if we keep BatchTopK fixed and vary and the training batch size, how much do the learned features and results change?

This is a great question. Just to be explicit about BatchTopK and what we’ve done in our experiments:

  • BatchTopK selects the top Bk activations across a batch, so the selection threshold for an example depends on the other examples in the training batch, and consequently, training batch size may affect the learned representation.
  • Our experiments use batch sizes between 256 and 1,024, but we do not perform controlled batch-size sensitivity analysis.
  • At inference, we use a fixed threshold calibrated from training data, so inference-time activations do not depend on the composition of the inference batch.

We do not expect results to change much (i.e. disappearance of representations or discovery of new representations), but we will test this explicitly and add our findings to the Appendix.


  1. The authors' statement regarding CEBRA is incorrect: CEBRA-time is not among the best-performing models in the Neural Latents Benchmark; CEBRA-behavior is. This may seem a trivial detail, but as it has been shown across several works at this point, they can differ significantly in terms of downstream performance.

Sorry for the confusion! Of note, we mention CEBRA generally, not CEBRA-behavior nor CEBRA-time explicitly. What we meant is that CEBRA-time and LangevinFlow are the best, unsupervised, current top-performing contrastive and sequential variational methods, respectively, on Neural Latents Benchmark. We specify unsupervised as this a key constraint that makes them fair for comparison with NLDisco. We will make this more clear in the text.

  1. Even though they explain what dashed lines represent in Fig. 3 in the caption, it should be put in the figure legend too.

Thanks for this point! We will update the figure legend to make this clear.


We hope this full response to each of your critiques in the form of clarifications and suggested edits would significantly impact your rating, and please do let us know any specific suggestions you may have in mind regarding these points! Thanks again for the thorough review!

Comments on the paper

3 comments
AuthorsOpenReview ↗

Urgent: Request for 12–24-hour rebuttal extension due to university network outage

Dear all, and specifically ICLR Area & Program Chairs,

I’m writing to request a short extension (12-24 hours) to the rebuttal period for this paper, “A pipeline for interpretable neural latent discovery”, submission number 49198.

An unexpected network outage at my university starting approximately 7 hours ago has made it impossible to access relevant data, model checkpoints, and other research files we need to complete our response -- our entire HPC is inaccessible. We've received an estimate that access will be restored in around 10 hours (10:00 AM November 19 UTC). This has prevented us from completing the final analyses and upload the final manuscript needed to complete our responses to the reviewers, who have been very helpful in their communication throughout this period.

Would it be possible to extend our deadline by 12–24 hours, ideally until 12:00 PM November 20 UTC? We can provide the outage notice and evidence separately to maintain anonymization.

Thank you for considering our request.

AuthorsOpenReview ↗

Final Revision and Response to Reviewers

Thank you all for the very helpful discussions that have significantly strengthened the paper during this rebuttal period. We have now uploaded our final revision, and we are happy and proud to have addressed, in some form, every single weakness raised across the reviews. Below we summarize the 13 changes we made in doing so, and where to find them. Reviewer IDs identify the reviews that motivated each change.

  1. ST-NDT baseline (9Vz6, YXrQ; Sec. 4.2, Fig. 4, Table 1, App. 6.4.2). We trained and compared ST-NDT on the macaque dataset alongside CEBRA-Time and LangevinFlow, adding feature scores, decoding, and training-time comparisons.
  2. Multi-session models (9Vz6; Sec. 3, 4.3, App. 6.4.7). We updated the SED model architecture to include shared and session-specific latents to support multi-session training. We show both fine-tuning and frozen zero-shot applications of extending a model trained on one set of sessions to a new set of sessions.
  3. End-to-end feature discovery (8ZX3, YXrQ; Sec. 4.3, App. 6.4.6, Tables S7–S9). We compare NLDisco with four other methods for end-to-end feature discovery through two assessments: (1) recovery of 12 prespecified features; (2) blinded AI raters’ interpretations of randomly sampled latents, validated on held-out data.
  4. Finding neural representations corresponding to environmental features (9Vz6; Sec. 4.3, App. 6.4.11). A model trained without stimulus labels on Allen visual cortex data yields latents tuned to Gabor position, grating temporal frequency, an image, and a movie segment, with held-out AUROC 0.77–0.97.
  5. Batch-size sensitivity analysis (9Vz6; App. 6.4.8). Across batch sizes 512, 1024, and 2048 at fixed k, the reported features remain recoverable; the analysis also reports latent stability, reconstruction, and dead latents.
  6. Held-out evaluation for validation of a latent's representation (9Vz6; Sec. 4.3, App. 6.4.9). We retrain on 80% of the data, choose each feature’s latent on that portion, and evaluate its interpretation on the held-out 20% for four macaque and four mouse features.
  7. Feature-discovery workflow (8ZX3, YXrQ; Sec. 3, App. 6.3.5). We explain how researchers use dashboards and AI agents to inspect latents, refine candidate descriptions, and trace source activity.
  8. Dictionary-size sensitivity analysis (8ZX3; App. 6.4.10). We compare feature scores, latent reappearance, dead latents, near-duplicates, and training time across dictionary widths 128–1024.
  9. Activation-threshold sensitivity analysis for defining a single latent from other LVMs (8ZX3; Sec. 4.2, App. 6.4.3). We vary the activation rate used to threshold baseline latents and report coverage, specificity, and selectivity across rates, including matched-coverage and best-threshold comparisons.
  10. Neural reconstruction benchmark (YXrQ; Sec. 4.2, App. 6.4.4, Table S5). A shared Poisson readout predicts held-out spike counts from each representation: NLDisco and LangevinFlow are similar (0.156 and 0.159 bits/spike), CEBRA-Time scores 0.115, and ST-NDT scores 0.253. We also compare the models’ own decoders.
  11. Updating "Related work" (9Vz6; Sec. 2, App. 6.2). We discuss switching dynamical systems, PSID, DPAD, DFINE, pi-VAE, POYO, and NDT2, distinguishing their dynamical, manifold, and transfer objectives from individual sparse-feature discovery.
  12. CEBRA and NLB clarification (9Vz6; Sec. 4.2). We thank you for the correction: NLB includes supervised CEBRA-Behavior, not CEBRA-Time, and our comparison now makes this clear. We also added a comparison to ST-NDT upon request.
  13. Figure 3 legend (9Vz6; Fig. 3). We added the meaning of the dashed curves to the figure legend itself.

We were able to finish our rebuttal analysis and submit before the deadline despite this network outage, so this is no longer relevant, but thank you for any consideration given :)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.