- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper suggests to train an encoding model with linear readout and then to use latent features shared across animals and neurons to train a sparse autoencoder. Than the features from sparse autoencoders are mapped into interpretable rules using some behavioral and stimuli information and an LLM.
Critical strengths
- Relevant problem - it is very true that in the classic neuroscience setup the feature descriptions and neuron selectivities are paired and neuronal selecties might not be linearly predictable because the feature description was not formulated precisely.
- Testing on heldout animals- testing on heldout animals is a nice way to test if the rules generalize.
- Empirical improvement - They show that in the selected datasets the new rules make the neuronal activity more linearly predictable.
Critical weaknesses
- Linearity assumption: Thought the text the paper emphasised that the responses should be linearly predictable from the encoder features, which is not exactly correct, since most of the models would use a non-linearity after a dot-product with the readout to avoid negative responses (as a living organism cannot have this)
- Error bars While the paper reports some confidence intervals across data bootstrapping in the text - fig 3 and 4 contain no error bars. SAE are known to not recover exactly the same concepts across several model refits (https://arxiv.org/abs/2501.16615) and the paper does not estimate the stability of SAE reconstructions apart from one sentence (lines 360-361).
- Clarity: Its only stated in the appendix what are the 318 input features are and its not clear for me how they were selected. Same for the amount of SAE bits - its not clear why its eight of them (I have not found an ablation in the appendix). Its also not clear for me how exactly the LLM proposed input-informed features - what was the specific propmt?
- AI-assisted diagrams require polishing - for example for Fig 1 - small details are missing, like in scalar features only one probably should be colored with green, no? Also it feels a bit overloaded - like why the contrasts are different in B but not in C and D? For Fig2A - the first panel implies that the 318 inputs are somehow transformed to neurons while in fact they are just paired (i dont think we can make a causal claim if neuronal signal or behavioural features come first).
- Related work missing - there are a few works analyzing neuroscience data with AI, which is missed in the related work a) https://arxiv.org/pdf/2605.12485 - summarises most exciting stimuli for V1 neurons using VMLs b) https://www.biorxiv.org/content/10.1101/2025.11.12.688086v3 claims to discover tuning laws for V1 neurons using LLMs
- Proof of concept example and bigger dataset - Since the paper misses the error bar and the improvement is noticeable but small - a paper would benefit a lot from a synthetic toy example as a validation / proof of concept for their pipeline (numerically simulated toy example). The current example would benefit from the estimated trial-to-trial noise ceiling - to contextualize the R^2 scores. I would also inspire the authors to repeat their pipeline on a bigger dataset since right now its ~550 neurons and ~160 rules, which is like 3 neurons per rule, so the rules might not be general enough.
Minor:
- the choice of digits to be reported seems to be arbitrary - for example in line 376 4 digits are reported despite the 3rd one is already quite different
Questions for authors
- Linearity assumption: - In equation (1) the function is a non-linearity, isnt it? Normally, this would be a non-linearity which would make the responses strictly non-negative. But then in equation (3) the is dropped and responses could be predicted negative which does not really happen in a living organism. Should there still be a non-linearity in eq. (3)?
- SAE stability evaluation: In lines 360-361 its said that 63 out of 164 rules got nice MCC scores across 20 SAE initializations. Just to make sure - were this the same rules or is it just at least 63 rules for each SAE initialization got . How similar are the rest 101 features across SAE initializations? (e.g. how many of the learnt features vary across initializations)? How stable is SAE across re-fits?
- Size-fold evaluation: Section 3.3 - Why do we need the 4-size evaluation fold? Where does size come from - isn't it cross-validation of ordering and amount of K needed? A schematics would help for this paragraph as its writing is confusing at least for me.
- SAE technicalities: In line 212 it says that "A sparse autoencoder (SAE) is then trained to reconstruct normalized hidden vectors." - why do we need to normalize them? In line 216 it says "We use 256 atoms and at most eight positive coefficients" - why only positive and not negative and select by the amplitude? neurons can be selectively suppressed by sth?
- Clarity 1: What is in line 165 - mutual information?
- Clarity 2: Is formula 2 basically normalized MSE? like 1 - R^2?
- Clarity 3: what exactly is outer-training partition and inner-training partition? 15 and 7 other mice or what?
- Were LLM suggested featured needed?: For the LLM suggested features - how many of the LLM proposed features survived? what is the statistical significance of these? what are their semantic?
- Scalability: How scalable is this pipeline if I have much more base features to select from? What is the complexity in notation based on the amount of potential features for rule construction and rule selection?
- SAE dimensionality ablation: How has eight dimensionality for SAE been chosen? Have you run an ablation for more or less active features?
- Multicollinearity: Do I get it right that many of the 318 used for rule construction are strongly correlated? What is you would just remove strongly correlated features from the proposals?
9 comments
Response to Reviewer 2cLA (2/5)
[W3] Clarity: 318 inputs, k = 8, LLM prompt
Response, 318 inputs: We did not select these inputs. They are exactly the encoder design matrix of Nogueira et al. (2023), rebuilt with their public code; all 70 sensory columns matched their builder at r = 1.000. The matrix has 14 channels over five 0.1-s lags plus eight current/previous-trial variables expanded over 31 time bins: (Appendix A.2). The 14 channels are signed and absolute lick rate, and contact-start rate, contact angle, and angular position for whiskers C0–C3. Keeping their basis is deliberate: the question is whether the nonlinearity reported for this basis shrinks when features are added. We will state this in Section 3.
Response, prompts: In the submitted version, the 122 input-informed features (Table S1, Appendix B.1) were not generated with a single standardized prompt. For this response, we wrote a standardized prompt that generates this vocabulary and measured how much its output varies. The prompt gives the LLM only the input layout and the design principle: "an exhaustive enumeration of low-order aggregates over the whisker and lag structure", including counts of positive entries. In five independent runs it regenerated 106 or 107 of the 122 features every time; the features it missed were the thresholded whisker counts. The variation was equally small downstream. We replaced the 122 features with the features generated in each run and refitted the linear baseline, which contains only the 318 inputs and the input-informed features (no rules and no atom-informed features). Held-out R² was 0.1107–0.1124 across the five runs, against 0.1079 with the original 122 features and 0.1005 with the inputs alone; each run also generated further aggregates beyond the 122.
The atom-informed proposals are generated by a prompt, which is in the supplementary code with its output schema and the per-atom evidence records (neural_repro/proposal_features.py, function prompt). We will add the standardized prompt in full, with these results, to Appendix B.5 and the supplementary code.
Response, k = 8 (full answer in [Q10/W3] below): It was set during development, not tuned. We ran the requested ablation: the rules exceed matched random rules at every from 4 to 64 (to be reported in Appendix A.5).
[W4] Figures
Response: We will revise both figures.
- Figure 1. Both and contribute to the single scalar feature in Fig. 1B. We will show their arrows converging on one node inside a green box. Following the convention in C, we will also draw the two upstream coordinates in D as two nodes inside a purple box. The difference in contour contrast in B has a specific meaning: only determines the response. We will explain in the caption that the faint contours mark the second coordinate, which can vary at fixed without changing the response.
- Figure 2A. The inputs and the neural recordings are paired measurements. We will remove the arrow between them and state this in the caption.
[W5] Related work
Response: Thank you; both are relevant. In the revision, Section 5 will cite Tilbury et al. alongside LLM-guided program evolution and Lad et al. alongside language descriptions of cortical selectivity, and Appendix F will state the difference: they characterize responses to a known stimulus variable (orientation) or image set, validated by single-cell fit or in silico synthesis, whereas our rules are executable descriptions of multivariate inputs tested for linear compatibility on held-out recorded sessions.
Response to Reviewer 2cLA (3/5)
[W6] Proof of concept, noise ceiling, bigger dataset
Response, synthetic toy example: We agree that a simulated example with known ground truth is a valuable validation of the pipeline. We are preparing a numerically simulated example in which we generate the responses from features that we define ourselves, so that we can test directly whether the pipeline recovers them. We will report it in a second response and include it in the revision.
Response, noise ceiling: We estimated the noise ceiling at 0.36 (session-bootstrap 95% CI 0.31–0.40). Relative to it, feature discovery raises linear prediction from 30% to 35% of explainable variance, a gain of 4.9 percentage points (CI 4.1–5.8) that is far from negligible on that scale. The animals whisk actively, so no two trials repeat the same input and a repeat-trial ceiling is not available. We therefore used a split-half estimate within each 0.1-s bin: the bin is cut into interleaved 10-ms slices, alternate slices give two half-counts N₁ and N₂, and the ceiling is 4 Cov(N₁, N₂) / Var(N₁ + N₂), computed on the same held-out observations as the R² values. Slice widths of 5, 25 and 50 ms and a Poisson reference give gains of 4.6 to 5.3 points. The revision will report the analysis in Section 4.2 and Appendix G (sensitivity in Table G1, neuron-level diagnostics in Table G2).
Response, bigger dataset: We are applying the same pipeline to additional data beyond these recordings. We will report the results in the second response and include them in the revision. Within the present data, the rules already generalize beyond the sessions used to find them: rules discovered in 15 sessions and then held fixed improved prediction in all seven excluded sessions (190 neurons; R² from 0.102 to 0.119; Section 4.3). Every rule is selected for its contribution to predicting all 575 neurons through session-specific linear readouts, so each is supported by the whole population.
Minor
Response: We will follow, and state in Appendix C.1, a rule tied to the uncertainty: R² levels are given to three decimals in the main text, because their session-bootstrap CIs span about ±0.01 (line 376 will read 0.108 to 0.125), and paired differences with their CIs to four decimals, because paired CIs are several times narrower. Where two levels under comparison differ only in the fourth decimal, we give four.
[Q1] Linearity assumption
Response: See [W1/Q1] above.
[Q2] SAE stability evaluation
Response: See [W2/Q2] above.
[Q3] Size-fold evaluation
Response: Yes. "Size" is the number of selected rules, , and the size-evaluation folds are the cross-validation that chooses it. Forward selection orders the rules, and choosing on the trials that produced the ordering would be optimistic. We therefore split the selection trials into four trial-grouped folds, matching the four-fold splits used elsewhere. For each fold, the ordering is computed on the other three, and every prefix is scored on the held-out fold. is the maximum of the averaged curve. The ordering is then refitted on all selection trials, and its first rules are kept (Appendix B.3). The revision will add a schematic of this procedure (Figure B1, Appendix B.3), and Section 3.3 will state that these folds choose .
[Q4] SAE technicalities
Response: The SAE decomposes the direction of the hidden state rather than its size. Row norms of vary about fivefold across samples (1st–99th percentile 0.83–4.24), so without normalization high-norm samples would dominate the squared reconstruction loss. Downstream, only which atoms are active is used (binary rule targets), so magnitude plays no role there.
Nonnegative codes are the standard TopK SAE form (ReLU, then the largest entries), and they do not prevent suppression: atoms describe the shared representation, and a neuron is suppressed through negative readout weights on the selected features (Eq. 3). The reviewer is right that sign matters at the atom level: a bipolar direction would be split into two atoms that become separate rule targets. In our dictionary this is rare. Of 15,400 pairs of live atoms, the most opposite has decoder cosine −0.65 and only 5 fall below −0.5. Atoms in these pairs almost never co-fire (≤0.3% overlap), yet their rules overlap rather than mirror each other (rule MCC +0.29 to +0.44). Signed codes with selection by magnitude are a natural extension, and the Discussion will note it.
[Q5] Clarity 1
Response: Yes, at line 165 is mutual information; the revision will define it there.
[Q6] Clarity 2
Response: Yes. Eq. 2 is the population average over neurons of for the best affine predictor, i.e. one minus the normalized MSE, which is the population R². Held-out R² is its finite-sample estimate.
Response to Reviewer 2cLA (1/5)
We thank the reviewer for the careful reading. We have carried out additional analyses in response to the comments and report them below. The synthetic proof of concept and the application to additional data (W6) are in progress and will follow in a second response. The revised manuscript, with all changes, will be uploaded in a single update.
[W1] Linearity assumption
Response [W1/Q1]: Firing rates are nonnegative. Equation 3 predicts each neuron's standardized firing rate, so zero on this scale corresponds to that neuron's mean rate. A negative output predicts activity below the mean. Applying a nonnegative output function on this scale would prevent the model from predicting those decreases. We will state the target scale beside Eq. 3.
The gain from the discovered features also holds when the readout is constrained to be nonnegative or monotone. We replayed the held-out evaluation with a rectified readout on the firing-rate scale and with a monotone output function fitted without test data. Only 3.5% of the affine predictions fell below a firing rate of zero, and the gain remained under both readouts:
The revision will report this analysis in Appendix C.6.
Equation 1 allows two sources of nonlinearity. The upstream transformation gn maps the measured inputs x to latent features zn=gn(x). The monotone response function ϕn then acts on a weighted sum of those features plus a bias. We will label the path from x to z in Fig. 1A with gn and explain in the caption that the figure uses an identity response function to show how the same response can be linear in one description and nonlinear in another.
Equation 3 puts linear compatibility (Eq. 2) to use. The encoder can learn nonlinear combinations of the inputs, and its affine readout asks how well a weighted sum plus a bias predicts the response from that representation. We use the same criterion to evaluate the sensory rules: we fit an affine readout to the original description and another to the description augmented with the discovered rules. The gain on held-out trials therefore tells us how much more of the S1 response a linear readout can explain once those sensory conditions are made explicit.
[W2] Error bars and SAE stability
Response [W2], error bars in Figures 3–4: We computed 95% confidence intervals for both figures and will add them in the revision, using the same session bootstrap as the CIs in the text (10,000 draws over 22 sessions, Appendix C.1).
The revised captions will state the CI method.
Response [W2/Q2], SAE stability: The same 63 rules: the 164 rules are fixed, and each was scored against each of 20 new SAE dictionaries; 63 reached best-match MCC ≥ 0.40 in all 20 runs. Of the remaining 101, 35 reached it in 15–19 runs, 16 in 10–14, 12 in 5–9, and 38 in 0–4 (Appendix C.2, Figure S1). Pooled median best-match MCC was 0.437, versus 0.276 for matched random rules and 0.120 under trial-block permutation. The atom directions themselves do vary across seeds (median matched decoder cosine 0.63 at 256 atoms; Appendix A.5), consistent with the paper the reviewer cites. Our method does not depend on recovering identical atoms. Rules are frozen and kept for their contribution to neural prediction, and correspondence to atoms only characterizes them. Seed-to-seed variation in SAE atoms therefore does not undermine the discovered features: 98 of the 164 rules find a matching atom in at least 15 of 20 independently trained dictionaries, their agreement is well above that of matched random rules, and the prediction results rest on the frozen rules, not on any particular set of atoms.