acceptodds
Under review as a conference paper at ICLR 2027

Demixing Task-Conditioned Shared and Private Neural Subspaces with Temporal Attention

Abstract

Single-region neural populations often mix sensory and decision-related signals, making it hard to tell which activity reflects sensory evidence, which reflects emerging choice, and when each becomes useful on a single trial. We introduce the Time-varying Subspace Autoencoder (TSA), a Poisson autoencoding model with subspace-specific temporal attention readouts. TSA decomposes time-varying neural population activity into stimulus-private, choice-private, and shared latent subspaces, and uses separate temporal attention heads to determine when each subspace contributes to stimulus and choice decoding. On synthetic data, TSA recovers the ground-truth temporal ordering and private/shared latent structure; on barrel-cortex recordings from a perceptual decision-making task, it yields an interpretable private/shared organization with single-trial temporal readout patterns in which stimulus-related signals typically precede choice-related signals, while reversed ordering is enriched on error trials. Together, these results position TSA as a tool for studying how sensory and decision signals are distributed across shared and private neural subspaces and how their readout evolves over time.

Primary area:
applications to neuroscience & cognitive science
Keywords:
neuroscience, perceptual decision-making

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_fQyvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper uses a Poisson autoencoder with latent partition into stimulus-private, choice-private, and shared subspaces; causal convolutional encoder; subspace-specific temporal attention gates; The strongest defensible novelty is therefore the specific task-private/shared + learned temporal-gating construction. It is validated on synthetic data with known ground truth and barrel-cortex recordings from a mouse whisker task; main claims are highest held-out stimulus decoding, emergent alignment of stimulus readout with whisker contact, and stimulus-before-choice ordering.

Critical strengths
  • The scientific question is the best thing about this paper. Asking not just whether stimulus and choice are decodable, but from which subspace and when on each trial, is a meaningful step beyond dPCA-style methods. Nobody has done this within-region private/shared split with single-trial temporal readout weights simultaneously before, and the related work section makes a convincing case for that gap.

  • The ablation suite is genuinely thorough — layout search, scorer variants, label shuffles, loss term ablations, reconstruction weight sweep.

Critical weaknesses
  1. In figure 3B related to real data results, the stimulus-private block decodes stimulus at 0.747 and choice at 0.690. The choice-private block does better (0.877 vs 0.637). The selectivity of the stimulus-private block is very weak. The authors acknowledge this but frame it as "expected" because choice is more decodable and the variables are correlated. However this seems a weakness in the core claim about private subspaces.

  2. Synthetic benchmark comparison seems insufficient. For a method claiming to identify which subspace is read out and when, you would want to know whether the model correctly identifies choice-first ordering when ground truth is choice-first, or whether it always biases toward stimulus-first due to the loss weighting asymmetry. Also authors do not compare against TAME-GP or any external method with known ground-truth recovery, which would be the natural use of having a synthetic benchmark.

  3. A simple causal baseline comparison would help clarify the results. Appendix F.1 already reveals that raw spikes carry the stimulus-before-choice ordering without any subspace model. A causal convolutional model with the identical attention pooling mechanism applied directly to raw spikes — no encoder, no decoder, no latent partition — would tell you how much of TSA's temporal performance comes from the attention gate architecture versus from the subspace decomposition. Without this, you cannot attribute the timing result to the scientific contribution of the paper.

  4. Table 7 explicitly shows that the 23/23/24 layout was selected because it has "the highest cross-seed readout reproducibility of any layout... and the highest correlation between the stimulus readout and whisker contact." The paper states: "The balanced allocation is therefore justified on the grounds... a readout that is stable across initializations and aligned to the sensory event." Contact alignment (ρstim,contact) was directly used to choose which layout to report. It is then also the primary external validation signal claimed in Section 5.3 and Table 1. This is circular. The independence claim of the contact alignment result depends on contact having no role in model decisions, but it was used to choose the latent layout.

  5. inconsistency: Main text Section 4 says "stimulus attention peaked before choice attention on 0.949 ± 0.006 of trials." Appendix Table 6 for the same experiment reports TSA (full) Stim-first = 0.694. These both describe synthetic data with 9 runs (3 dataset × 3 model seeds). That is a 0.255 difference on the key timing metric, could you please clarify this discrepancy?

Minor weakness:

  • Please number the equations so they can be referred to.

  • Please improve the caption of figure 2 for clarity. The legend used in second column also causes confusion, do you mean latent instead of neuron?

Questions for authors
  • The denominator in the attention weight formula sums sigmoid scores over all T time bins including τ > t. This means α_i^r(t) requires knowing future scores at inference time. Can you explain how strict causality is maintained under this normalization?
  1. The choice of 70-dimension latent with 14 neurons in real-data seems very unusual, have authors tried low dimensional latent and did not get better results?

  2. Were you aware of “Multi-Integration of Labels Across Categories for Component Identification in Multi-trial Time Series” (ICML 2026), which also partitions multi-trial neural data by label categories with per-trial temporal traces? How would you distinguish TSA's contribution from this framework given that the shared/private category structure and per-trial temporal modeling are both present there?

  3. there is no mechanism to capture task-irrelevant variance. All latent dimensions are partitioned into stimulus-private, choice-private, or shared. The reconstruction term forces the model to use all three blocks to explain the spikes, meaning task-irrelevant activity gets absorbed into whichever block has capacity. This contaminates the decomposition. Does the absence of a task-irrelevant subspace affect the interpretability of the shared block? neural activity in the barrel cortex is not fully explained by stimulus and choice variables alone. What happens to movement-related or other non-task variance in the current model?

  4. why was TAME-GP not included as a baseline, given it is described as the closest prior work?

3 comments

Thank you for a sharp review. You suspected the gate, not the partition, does much of the work; that is largely right, and the paper now says so (General Responses 3–5).

W1. The stimulus-private block decodes stimulus 0.747 and choice 0.690: weak selectivity. Agreed. We checked what the split really does, and why it cannot do more.

  • What it does: on the main session it makes the stimulus-private block prefer stimulus (0.73 vs 0.66; without separation 0.79 vs 0.89). In new sessions it does not; it only makes the block follow the choice less on error trials (General Response 5/5).
  • Why not more: a version of the model with exactly private blocks (the other variable projected out) leaves them nearly empty (stimulus-private → stimulus 0.53), because stimulus and choice information overlap in these neurons (General Response 3/5).

Paper change: "expected" is withdrawn; the paper reports the full block-by-variable table and where a preference holds.

W2a. Does TSA recover a choice-first order when that is the truth? Yes. On simulated data with the true order reversed, TSA reports stimulus-first on only 0.13 of trials (0.73–0.83 when the truth is stimulus-first). We also swapped the roles of stimulus and choice inside the model (the losses written for stimulus act on choice and vice versa). In a separate simulator (calibrated on all trials rather than training trials only), the recovered order still follows the true order (0.04 stimulus-first when the truth is choice-first, 0.94 when it is stimulus-first), so the order comes from the data, not from the losses. TSA does under-recover the order (about 0.8 where the truth is 1).

W2b / Q4. No comparison with TAME-GP. Added, with both our reimplementation and the authors' official code, tuned on training trials only.

TAME-GP (authors' code)TSA (23/23/24 version)
simulated: fraction of trials with the true order0.890.73–0.83
simulated: per-trial timing, correlation with truth0.100.71–0.82
real data: stimulus / choice accuracy0.60 / 0.950.75 / 0.90

TAME-GP is better at choice and at the order score. That score only asks which peak comes first, so it can be high even when the peaks are in the wrong place; TAME-GP's timing correlation of 0.10 shows it does not locate the events. TSA is much better at stimulus and at per-trial timing. We also corrected our description of TAME-GP (its private/shared split is across brain areas).

W3. Apply the same causal attention directly to raw spikes. Done. As you expected, the gate explains most of the decoding and timing:

main session (the same 20 seeds for both rows; 23/23/24 version)stimulus acc.choice acc.correlation of stimulus attention with touch times
TSA0.740.900.58
same attention on raw spikes0.720.900.50

On simulated data the raw-spike gate also recovers the per-trial timing. What TSA adds over it: a latent model that also reconstructs the spikes, and private blocks that prefer their own variable on the main session (General Response 5/5). A simpler model with two gates on one shared latent and no subspaces also decodes as well (General Response 3/5), so we do not claim a decoding or timing advantage for the partition. Its role is to make the block names mean something, clearly on the main session and only partly in new sessions (General Response 5/5).

W4. Contact alignment chose the layout and then validated it: circular. Yes, it was. Now:

  1. no test data or contact is used for any choice (General Response 2/5);
  2. contact alignment is tested on 17 new sessions where it was never used: stimulus attention follows touch in all 17;
  3. the layout is no longer argued from contact: all layouts of 36+ dimensions give the same results in the new sessions (General Response 3/5).

We also note that the choice attention follows touch about as well, so contact alignment is no longer presented as evidence for a stimulus subspace.

W5. Stimulus-first 0.949 (Section 4) vs 0.694 (Table 6). The two numbers came from different models and code versions (General Response 2/5). Both are removed; the redone benchmark gives 0.73–0.83 (DZq9 response, Q3).

Minor. Equations are numbered. The Figure 2 legend shows observed neurons grouped by the factor they load on; Figure 2 is replaced together with the old synthetic results, and the new figure labels this explicitly.

Q0. The normalizer sums over future bins; how is causality kept? It is not, for the normalized weight; our wording was wrong. What stays causal: each bin's gate value and latent use only past spikes, and the normalizer divides all bins of a trial by the same number, so it cannot change which bin is highest; peak times and orderings are unchanged. The paper's causality sentence is corrected.

Q1. A 70-d latent with 14 neurons is unusual; did you try low-dimensional latents? Yes. The private blocks can be small, but the whole latent must not be too small (17 new sessions):

latentstimulus acc.run-to-run agreement, stimulus attention
6/6/6 (18-d)0.690.31
12/12/12 (36-d)0.680.34
23/23/24 (70-d, submitted)0.670.43
12/12/46 and 2/2/66 (70-d)0.680.43–0.44
20/20/20 + 10-d null (revised)0.680.41

Decoding is the same at every size. With 18 or 36 dimensions, the attention curves differ more between training runs (18-d: lower in all 17 sessions; 36-d: in 14; on the main session the stimulus attention barely repeats, 0.12 vs 0.41). A likely reason: the spikes of each time bin are reconstructed from that bin's latent alone, so the latent must also hold recent history; with few dimensions, little room is left for the task signals.

Q2. How does TSA differ from MILCCI (ICML 2026)? Thank you; we will cite it. Both use per-trial labels and time courses, but:

MILCCITSA
labels areinputs that select componentstargets; a test trial is encoded without its labels
shared structurenone between categoriesan explicit block shared by stimulus and choice
time coursesmoothed over the whole trialcausal per-bin readout weights

Q3. No task-irrelevant subspace: where does movement-related variance go? Thank you; we adopted your suggestion. TSA (revised) has a 10-d null block (20/20/20 + 10). It is used only to reconstruct the spikes, and during training we repeatedly remove from it the directions that separate the two stimuli and the two choices.

18 sessionsresult
null block decodes stimulus / choice, time-averaged0.50 / 0.50 in every session (chance)
the same, pooled by the attention gates0.53–0.56 (new sessions), vs 0.60–0.77 for the task blocks
decoding, run-to-run agreement, effect of the splitunchanged vs 23/23/24 (General Response 3/5)

Movement signals still mix into the task blocks: in the 23/23/24 version almost all of the latent's prediction of whisking amplitude (R² 0.41) went through the shared block, and removing licking and whisking from the spikes lowers choice decoding from 0.77 to 0.54. The null block holds mostly task-free activity, but it does not separate movement from the task signals. The paper states this limitation.

Reviewer_Dyo8OpenReview ↗
Rating
2(−2)
Confidence
3
Informativeness
2(−1)
Presentation
3
Soundness
1(−2)
Read review
Paper summary

The authors designed a new decoding method and an auto-encoding framework, Time-varying Subspace Autoencoder (TSA), which demixes stimulus- and choice-related signals from the brain activity. The method is based on an authoencoding model which assumes Poisson statistics of neural activity, and has temporal attention readouts that are specific to either the stimulus, the choice and the shared readout. The method minimises a composite loss function, with elements of the loss being introduced sequentially during training. The method is tested on synthetic data (with known ground truth) as well as on real data from the mouse sensory cortex, and mostly performs similarly or sometimes better than other state-of-the-art methods. Its main advantage is the interpretability of results with respect to timing of stimulus- and choice-related information as it can be decoded from the neural activity.

Critical strengths

The paper is written very clearly and is a pleasure to read. It proposes a potentially interesting method that can be useful to other scientists to study mixed information in neural data (and mixed representations are practically omnipresent in the brain).

Critical weaknesses

After reading other reviews of this submission, I have verified the paper again and found the problems already reported by other reviewers. I have missed most of these issues in the first review, hence a significant update to my review, in particular the Weaknesses section. Because of several serious weaknesses, I had to lower my general score.

Major:

  1. The model is tested on insufficient data with only two recording sessions from two animals, each having only a small number of neurons.
  2. Lack of generalization of preferential task selectivity of the private subspaces, acknowledged in appendix D, limits the significance of results and questions the credibility of the method.
  3. There are several inconsistencies between the Appendix and the main paper, in particular, in reported results about stimulus-before-choice fraction, stimulus- and choice-private recovery and shared latent recovery.
  4. There is a potential issue with the Equation in the section 3.3 where the denominator (normalization) depend on the future neural activity, making the attention weights non-causal. This is in contradiction with the main text [lines 198-199] that claim that attention weights are causal.

Minor:

Line 83: there seems to be a typo Line 134: It remans unclear how is attention in TSA used in a “narrower sense” Line 169: It is unclear why the authors used 23 /23/24 dimensions (which seems like a lot) and what is a layout in this context.

Questions for authors

(1) What does Poisson assumption entail? Does the neural activity need to satisfy a homogeneous Poisson process or is it enough that it behaves as a non-homogeneous Poisson? How well does prediction work if neural activity deviates from a Poisson process? [clarification]

(2) The loss function has several components which are taken into account sequentially during training. How did authors decide about the order in which different components of the loss are introduced? [clarification]

(3) If the angle between stimulus and choice subspaces was smaller, what would be the expected effect on decoding? The authors do not need to add this to the revision, it is a rather speculative question. [suggestion]

(4) Is there a difference in overall performance between behaviourally correct and error trials? [suggestion]

(5) Authors mention the issue with a small number of held-out trials that are also reversed. Did the authors evaluate their statistic on multiple cross-validated held-out blocks? Alternatively, a random split of trials into training and test set which is repeated many times could be a useful method to increase the statistical power of these results (as done in this study [1]). [clarification]

(6) The study in [1] also decodes binary stimulus and choice from a primary sensory area (in their case, monkey V1) while using the overlapping information between stimulus and choice. Similar to your study, [1] decodes the choice from single trials and retains time-dependent information from the data, while it also finds strong choice-related representation in a primary sensory area. Can authors comment, if they find it useful, on their results with respect to this study? [clarification]

(7) While the binary stimulus / choice makes sense for testing of the model, can authors comment about the possibility to extend the model beyond this binary setup? [suggestion for future work]

(8) Barrel cortex is a primary sensory area and it is interesting that the decoding of choice has a stronger Balanced accuracy than the decoding of stimulus. While I do not think that this necessarily indicates a problem with the method, I would like to ask the authors to comment on it. [clarification / discussion]

[1] Learning from invariants predicts upcoming behavioral choice from spiking activity in monkey V1; Veronika Koren, Ariana R. Andrei, Ming Hu, Valentin Dragoi, Klaus Obermayer bioRxiv 2020.01.10.901504; doi: https://doi.org/10.1101/2020.01.10.901504

2 comments

Thank you for the positive review. While answering all four reviews we found that checkpoints had been chosen on test trials and that the code did not match Figure 1; both are fixed (General Response 2/5), so some numbers below differ from the paper.

Line 83 typo. We found an inconsistent model name ("Time-varying" vs "Time-Varying") and fixed it. Please tell us if you meant something else.

Line 134, "narrower sense". Rewritten: TSA's attention is one weight per time bin and block, computed from recent spikes and used only to take a weighted average of that block over time. It is not query–key attention: time bins do not attend to each other.

Line 169, why 23/23/24 and what is a layout. A layout is how the latent dimensions are split into stimulus-private / choice-private / shared blocks. We compared six layouts on all 18 sessions (General Response 3/5): decoding is the same for all of them, and only a very small latent (6/6/6) makes attention less reproducible. The original 23/23/24 was partly chosen with contact alignment (General Response 2/5); TSA (revised) uses 20/20/20 plus a 10-d null block for task-unrelated activity (fQyv's suggestion), and gives the same results.

Q1. What does the Poisson assumption require? Only that counts are Poisson given the latent; the rate can change over time and across trials, so the data need not be a homogeneous Poisson process. The recorded counts are over-dispersed (Fano factor 1.28), but the model absorbs this through its trial-varying rates. In the 23/23/24 version, replacing Poisson with negative binomial or Gaussian likelihoods changes stimulus accuracy by ≤ 0.015 (within one seed s.d.), so we keep Poisson.

Q2. How was the order of the loss terms chosen? By one rule: first put information into the blocks, then remove the wrong information. Reconstruction learns the activity; decoding and the private/shared heads then put task information into the blocks; only after that do the separation terms remove the other variable. On the main session (23/23/24 version), moving the stage boundaries earlier or later leaves stimulus accuracy unchanged; starting with the separation terms lowers it by 0.03, and turning all terms on at once by 0.12. (The paper's Table 11 showed little effect because its checkpoints were selected on test trials, which hides poorly trained runs.)

Q3. What if the stimulus and choice subspaces had a smaller angle? Simulated sweep: decoding hardly changes (0.985 at 90° vs 0.958 at 0°), but the model can no longer tell which subspace carries what, and the stimulus-before-choice order disappears below about 15°.

Q4. Correct vs error trials? Decoding is higher on correct trials in all 17 new sessions for stimulus and in 16 for choice. On error trials stimulus decoding falls to chance (0.48), for TSA and for a decoder on raw spikes alike; choice stays decodable (0.64).

Q5. Repeated cross-validation for the reversed-ordering statistic? Done, and it changed our conclusion. Pooling repeated 5-fold CV over the 17 new sessions (~2,700 held-out trials), reversed-order trials are not more often errors; if anything they are slightly less often errors (20% vs 23%, p = 0.045). We withdraw the claim. Thank you.

Q6. Relation to Koren et al. (2020)? We ran their idea on our data: a stimulus decoder trained on correct trials (where stimulus = choice) reports the animal's choice on 62% of held-out error trials (more than half in 15 of the 17 sessions). So barrel cortex has the same stimulus–choice overlap they use in V1. Their method uses this overlap; TSA tries to separate it, and succeeds only partly (fully on the main session; in new sessions it only removes some choice from the stimulus-private block on error trials; General Response 5/5). We now cite and discuss the paper.

Q7. Beyond binary variables? Decoding extends directly: a simulated 3-valued stimulus is decoded at 0.82 (chance 0.33). The separation terms are written for binary labels; extending them is future work.

Q8. Why does choice decode better than stimulus in a sensory area? It is a property of the recordings: a simple decoder on the raw spikes also finds choice easier to decode than stimulus in 12 of the 17 sessions. Much of the choice signal is shared with licking and whisking: removing these behaviors from the spikes lowers choice decoding from 0.77 to 0.54 (decoder on spikes, average over sessions). The later rise of choice attention may therefore partly reflect the movements that report the choice. We therefore call it "choice-related", not "decision".

Reviewer_kYGxOpenReview ↗
Rating
1
Confidence
3(+1)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces the Time-varying Subspace Autoencoder (TSA) to separate stimulus-related and choice-related information within a single neural population while examining when these signals contribute to decoding. A causal convolutional encoder maps spike counts into three latent blocks: stimulus-private, choice-private, and shared. A pointwise decoder reconstructs neural activity under a Poisson objective, while separate temporal attention mechanisms pool the latent blocks for stimulus and choice classification. Reconstruction, auxiliary decoding, shared decoding, separation, and covariance penalties encourage the intended organization. The authors explicitly define “private” as preferential task information rather than exclusive information.

The model is evaluated on synthetic data with known latent factors and on two mouse barrel-cortex recording sessions. The paper reports improved stimulus decoding relative to the compared methods, alignment between stimulus attention and whisker contact, and an average stimulus-before-choice temporal ordering. Its main scientific contribution is the use of task-attributed attention trajectories as a descriptive account of single-trial processing. However, the evidence supporting that interpretation is substantially weaker than the evidence supporting its decoding performance.

Critical strengths
  1. Relevant research problem: The paper addresses the important challenge of separating stimulus-related, choice-related, and shared neural representations while identifying their temporal contributions to neural decoding
  2. Extensive experimental comparisons: The authors evaluate TSA against established methods, including dPCA, PSID, DPAD, SLDS, and rSLDS, and provide synthetic experiments, ablation studies, and additional validation on a second animal recording.
Critical weaknesses
  1. Limited experimental generalization: The real-data evaluation includes only two recording sessions from two animals, containing 14–17 neurons each. Such limited experimental diversity makes it difficult to establish whether the proposed framework generalizes to larger neural populations, different cognitive tasks, or other neuroscience datasets.

  2. Insufficient validation of task-specific subspace separation: Although the proposed framework aims to distinguish stimulus-private and choice-private representations, Appendix D (Table 8) shows that the intended selectivity does not generalize to the second animal. In some cases, the stimulus-private subspace predicts choice more accurately than the choice-private subspace. This weakens the claim that TSA consistently learns meaningful task-conditioned representations.

  3. Questionable temporal interpretability and stability: The authors interpret attention peaks as indicators of when task-specific neural information becomes prominent, but do not sufficiently validate whether these peaks correspond to actual contributions to prediction. Furthermore, Table 2 reports lower cross-seed ordering agreement for TSA (κ = 0.355) than for several simpler controls, questioning the robustness of its single-trial temporal interpretations.

  4. Potential evaluation bias and incomplete methodological justification: Appendix C.1 indicates that the latent configuration is selected partly using whisker-contact alignment on held-out trials, although contact alignment is subsequently presented as external validation. Additionally, the temporal attention normalization includes information from all time bins, contradicting the strict causality claim for normalized attention weights. These issues weaken the methodological rigor and independence of the reported validation.

Questions for authors
  1. Limited Generalization: How do the authors justify the generalizability of TSA when it is evaluated on only two recording sessions with 14–17 neurons each?

  2. Subspace Separation: How do the authors justify the effectiveness of task-specific subspace separation when the stimulus-private and choice-private representations fail to maintain their intended selectivity in the second animal (Appendix D, Table 8)?

  3. Temporal Interpretability: How can the authors establish reliable single-trial temporal interpretations when TSA exhibits lower cross-seed ordering agreement (κ = 0.355) than several simpler baselines (Table 2)?

  4. Evaluation Bias and Causality: How do the authors rule out selection bias from using held-out contact alignment for model selection and justify strict causality when attention normalization incorporates future time bins?

2 comments

Thank you. You were right on all four points (General Response 2/5 lists what we fixed).

Q1. Only two sessions with 14–17 neurons: how general is TSA? We now use 18 barrel-cortex sessions from 6 mice; 17 are new and never used to choose any setting.

17 new sessionsresult
stimulus decoded above chanceall 17 sessions
stimulus attention follows touchall 17 sessions
private blocks prefer their own variableno; separation only removes some choice from the stimulus-private block on error trials (14 of 17 sessions)
TSA decodes better than simple decodersno (withdrawn)

All 18 sessions come from one dataset and task; the paper now states this scope.

Q2. Stimulus- and choice-private blocks lose their selectivity in the second animal (Table 8). We agree the blocks are not cleanly selective, and we tested why.

  • What separation does achieve: on the main session each private block clearly prefers its own variable. In new sessions it only makes the stimulus-private block follow the animal's choice less on error trials, where stimulus and choice disagree (63% → 55% of error trials; 14 of 17 sessions). On all trials, separation lowers both variables in both private blocks about equally (full table in General Response 5/5).
  • Why cleaner selectivity is not possible here: we built a version of the model that makes the private blocks exactly free of the other variable (by projecting it out). On the main session its private blocks then carry almost nothing (stimulus-private → stimulus 0.53; choice-private → choice 0.60, held out), and all information moves to the shared block. In these recordings most stimulus information is mixed with choice information, so a block cannot be both informative and perfectly selective.
  • The second animal has the most correlated stimulus and choice of all 18 sessions (correlation of the stimulus and choice labels 0.70–0.76 across its three sessions; main session 0.23), the hardest case for any separation.

Paper change: we no longer call the private blocks "selective"; the paper reports the full block-by-variable table and where a preference holds.

Q3a. Do attention peaks correspond to actual contributions to prediction? Yes, for the model's readout. In the 23/23/24 version, deleting the 5 most-attended time bins lowers decoding (AUC) by 0.07–0.13, against 0.01–0.02 for 5 random bins (four layout and gate settings); deleting the least-attended bins has no effect.

Q3b. TSA's cross-seed ordering agreement (κ = 0.355) is lower than simpler baselines. You are right that models without separation agree more across seeds; our new measure confirms it. κ compares only the binary order of two peaks, so we now compare whole attention curves. We now measure directly how similar a trial's attention curve is across independent training runs (all 45 pairs of 10 seeds, averaged over trials):

run-to-run agreement, stimulus attention (training / test trials; chance 0.05–0.17 in new sessions)main session17 new sessions
TSA (revised)0.45 / 0.460.42 / 0.41
TSA 23/23/240.43 / 0.410.45 / 0.43
TSA 6/6/60.14 / 0.120.31 / 0.31
same model without separation0.62 / 0.590.60 / 0.60

Agreement is moderate, nearly the same on training and test trials, and higher than with a small latent. Models without separation agree more; we report this as a cost of separation (General Response 3/5). We no longer claim that the stimulus-vs-choice order of any single trial is reliable; the order is reported as a fraction over trials.

Q4a. Selection bias from using held-out contact alignment. You were right, and the leak was larger: checkpoints were also chosen on test trials (General Response 2/5). Now nothing uses test data or contact. Contact alignment is checked on 17 new sessions where it was never used, and holds in all 17.

Q4b. Causality when the normalization uses future bins. You are right: the normalized weight is not causal, and our sentence was wrong. Each bin's gate value and latent use only past spikes, and the normalizer divides all bins of a trial by the same number, so it cannot change which bin is highest: all peak times and orderings are unchanged. The sentence is corrected in the abstract, §3 and Figure 1.

Presentation. Part of the difficulty was real inconsistency (code ≠ Figure 1; tables from different runs). The revision uses one model, one protocol and one value per quantity, adds a "model at a glance" table and numbered equations, and states plainly what each loss term does.

Reviewer_DZq9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TSA, a method designed to disentangle signals related to stimuli, choices, and their interaction from mixed neural activity, while also identifying the timing at which these signals become functional. The authors validated the approach using both simulated data and experimental recordings from two mice. The results indicate that the model improves stimulus identification, with its temporal dynamics aligning with the process of whiskers making contact with objects; however, some findings could not be replicated in the second mouse.

Critical strengths

The research addresses a significant question: it seeks not only to determine what information is contained within neural activity but also to clarify when that information is utilized. The authors conducted a robust set of control experiments to support the method's efficacy in stimulus identification, and they transparently reported results that failed to replicate.

Critical weaknesses

The experimental scale is limited, and the failure to replicate key signal-disentanglement results in the second mouse leaves the generalizability of the method uncertain. Furthermore, the model selection process relied on the degree of alignment with whisker-contact events—a criterion subsequently used to demonstrate the method's validity—which weakens the argument. Additionally, the fact that a model focuses on a specific time point does not necessarily imply that the brain actually utilizes the corresponding information at that precise moment.

Questions for authors

Would the primary results still hold if the model were selected without reference to the test data or the whisker-contact records? What evidence suggests that the time points highlighted by the model truly correspond to the moments when the information is functionally employed? Moreover, why do the results of the simulation experiments differ between the main text and the appendix?

1 comment

Thank you. Your first question led us to audit the code (General Response 2/5).

Q1. Do the results hold if the model is selected without test data or whisker-contact records? Mostly yes. We retrained everything with no test-based selection (General Response 2/5) and tested on 17 new sessions where contact was never used.

resultwithout test or contact selection
private blocks prefer their own variablemain session only; in new sessions separation only removes some choice from the stimulus-private block on error trials (General Response 5/5)
stimulus attention follows whisker touchholds (all 17 new sessions)
stimulus attention peaks before choice attentionholds on the main session and for session averages; weak per trial in new sessions (General Response 4/5)
latent layoutno longer tuned on test or contact: all layouts of 36+ dimensions, and the new 20/20/20 + 10-d null model, give the same results in the 17 new sessions (General Response 3/5)
better stimulus decoding than controlswithdrawn

Q2. Do the highlighted time points show when the information is actually used? They show when the model's readout uses it; whether the brain uses it then would need perturbation experiments, and the paper now says so. Evidence for the narrower claim:

  • Deleting the 5 most-attended time bins lowers decoding (AUC) by 0.07–0.13, against 0.01–0.02 for 5 random bins and about 0 for the 5 least-attended bins (23/23/24 version, four layout and gate settings). The matched controls show the same, so this is a property of the attention readout.
  • Aligned to each trial's first whisker touch, stimulus attention rises within 0.2 s (main session, 23/23/24 version).
  • Attention properties change with the animal's behavior, in both training and held-out trials (11 behavior–attention pairs, General Response 4/5): e.g. after an error trial both private attentions are flatter.

Q3. Why do the simulation results differ between the main text and the appendix? They came from two different models and code versions, one of which was not saved (General Response 2/5). We removed both and redid the benchmark with the current model. The simulator now includes a realistic touch response measured from training trials.

simulated data (true order known; 23/23/24 version, two layouts)TSA
truth "stimulus first": fraction of trials where TSA finds stimulus first0.73–0.83
truth "choice first": fraction of trials where TSA finds stimulus first0.13 (correctly reversed)
per-trial timing: correlation between estimated and true time0.71–0.82 (TAME-GP: 0.10)

W1. Limited scale; the second mouse did not replicate. We now use 18 sessions from 6 mice (17 new; General Response 1/5). Decoding and touch alignment replicate; the private blocks' preference does not (Q1 table). The second mouse has the most correlated stimulus and choice labels of all 18 sessions (correlation 0.70–0.76 across its three sessions; main session 0.23), which makes the two hard to separate in any model (see also our reply to kYGx, 1/2).

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response (1/5): summary

Thank you for four careful reviews. They led us to audit our code, fix what was wrong, and test the model on much more data. In short:

you askedwhat we didresult
Were models selected using test data or contact? (DZq9, kYGx, fQyv)audited the code; retrained everything without any test-based selection (General Response 2/5)the code had kept checkpoints chosen on test trials; numbers are corrected (stimulus accuracy 0.81 → 0.75)
Two sessions are too few (DZq9, kYGx)added 17 sessions (18 in total, from 6 mice including the main-session mouse; all recorded neurons, 15–48 per session; the main session keeps the paper's 14); none was used to choose any settingdecoding and touch-aligned attention replicate; the effect of the split replicates only partly (General Responses 3–5)
Why 70 latent dimensions? No task-irrelevant subspace? (Dyo8, fQyv)added a 10-d null block (20/20/20 + 10); compared six layouts and two simpler models on all 18 sessions (General Response 3/5)the null block carries little task information; results do not depend on the layout, as long as the latent is not too small
Is it the gate or the partition? (fQyv)matched controls, the same gate on raw spikes, TAME-GPdecoding and timing come mostly from the gate; separation makes each private block prefer its own variable on the main session; in new sessions it only makes the stimulus-private block follow the animal's choice less on error trials (General Response 5/5)
Do attention peaks mean anything? (DZq9, kYGx)touch alignment in new sessions; deletion tests; attention vs behaviorattention follows whisker touch, drives the readout, and changes with behavior in both training and test trials (General Response 4/5)

What we withdraw: every number from test-selected checkpoints; "highest decoding" and better decoding than our own controls; the claim that trials where choice attention peaks before stimulus attention are more often errors; contact alignment as evidence for the partition.

Names used below. TSA (revised): the model we now use, 20/20/20 + 10-d null block. 23/23/24 version: the submitted layout retrained under the corrected protocol; it is used for analyses we did not re-run with the revised model. New sessions: the 17 added sessions. All accuracies are held-out balanced accuracy.

AuthorsOpenReview ↗

General Response (2/5): problems we found in our own code, and how we now train and test

Checking your questions about model selection, we audited the code. We found these problems and fixed all of them.

problemeffectfix
Checkpoints were chosen on test trials (the epoch with the best test accuracy was kept)stimulus accuracy was inflated by about 0.06 (0.81 → 0.75)no checkpoint selection (below)
The code did not implement Figure 1. Instead of pooling each block with its own gate, it joined the stimulus-private and shared blocks and pooled them with one extra gate.all TSA numbers, though only slightly (≤ 0.01)we implemented Figure 1 exactly and re-ran everything
Section 4 / Figure 2 came from an older code version that was not savedits numbers (e.g. 0.949 stimulus-first, fQyv W5) cannot be reproducedsynthetic benchmark redone
Contact alignment (how well stimulus attention lines up with whisker touches) helped choose settings such as the block sizescircular: the same measure chose and validated the model (kYGx, fQyv)17 new sessions where contact was never used
tables mixed runs with different settings and seed countsthe same quantity appeared with six different valuesone table, one protocol

How every number is now produced

  • Train on the training trials only; no early stopping on test data.
  • Use the average of the weights over the last 100 of 500 epochs.
  • Report the mean over seeds (10–30) and compare against matched controls: the same model with one part removed (no separation losses; no shared block; no private blocks; one undivided latent).
  • Data: the main session plus 17 new sessions from 6 mice from the same dataset and task (Rodgers, 2022), never used for any choice of settings. One of them is the Appendix D session.

The released code uses this protocol by default; a flag restores the submitted (test-selected) checkpoints for comparison.

AuthorsOpenReview ↗

General Response (3/5): the model we now use, and how consistent it is across 18 sessions

The model. Following fQyv (Q3), we add a 10-d null block for activity unrelated to the task: the latent is now 20/20/20 (stimulus-private / choice-private / shared) + 10 (null). The null block is used only to reconstruct the spikes. During training we repeatedly remove from it the directions that separate the two stimuli and the two choices. We chose this layout without looking at any data: equal task blocks, the paper's total of 70, and 10 dimensions for the null block. An ablation of the separation losses (5-fold cross-validation, 18 sessions) showed that one term, the role margin, has no effect, so it is removed; orthogonality, HSIC and the probe each matter.

Comparison. We trained this model, the submitted 23/23/24 layout, four other layouts and two simpler models on the main session and the 17 new sessions (10 seeds each, same data and splits). "Run-to-run agreement": we train each model with 10 different random seeds, correlate the attention curves of the same trial for every pair of seeds (45 pairs), and average over pairs and trials (1 = identical curves). Chance, from pairing a trial with a different random trial, is 0.05–0.24 in the new sessions (0.05 / 0.15 for TSA (revised); highest for the models without separation, whose curves share a common shape across trials). This is a stricter score than the paper's "reproducibility" and is not comparable to it.

17 new sessions (mean)

modelstimulus / choice acc.null block decodes stimulus / choicerun-to-run agreement, stimulus / choice attention
TSA (revised): 20/20/20 + 10 null0.68 / 0.770.50 / 0.50 (0.53–0.56 pooled by the gates)0.41 / 0.43
TSA 23/23/24 (submitted)0.67 / 0.77–0.43 / 0.42
TSA 12/12/46, 2/2/660.68 / 0.77–0.43–0.44 / 0.34–0.44
TSA 12/12/120.68 / 0.77–0.34 / 0.39
TSA 6/6/60.69 / 0.76–0.31 / 0.26
same model, no separation losses0.69 / 0.770.50 / 0.500.59 / 0.62
one shared latent with two gates, no subspaces0.70 / 0.77–0.50 / 0.54

What each block decodes is in General Response 5/5.

What this shows

  1. The null block carries little task information. Averaged over time it decodes stimulus and choice at exactly chance; this is guaranteed by the removal step, which equalizes the class means. Pooled by the attention gates it decodes 0.53–0.56, above chance but far below the task blocks (0.60–0.77). The other blocks decode as in the 23/23/24 layout.
  2. Decoding does not depend on the layout (0.67–0.70 / 0.76–0.77 for every model).
  3. The latent must not be too small. With fewer dimensions, the attention curves differ more from one training run to the next (stimulus attention: 6/6/6 0.31 and 12/12/12 0.34 vs 0.43 for 23/23/24; lower in 17 and 14 of the 17 sessions). The 70-d layouts all behave alike.
  4. Attention is as reproducible on test trials as on training trials (e.g. 0.42 vs 0.41 for TSA (revised)), so the model does not behave differently on trials it was fitted on.

A cost and a limit. (i) Cost: the models without separation give more reproducible attention than TSA (choice: 0.54–0.62 vs 0.43), also after subtracting chance: separation trades some stability for the changes described in General Response 5/5. (ii) Limit: in a variant of the model that projects the other variable out of each private block, making the blocks exactly free of it, they become nearly empty (main session: the stimulus-private block then decodes stimulus at only 0.53). Stimulus and choice information overlap in these neurons, so perfectly selective blocks are not possible in these data.

Analyses not re-run with TSA (revised) (simulations, TAME-GP, attention on raw spikes, deleting attended bins, alternative likelihoods and loss schedules) use the 23/23/24 version, which is within 0.03 of TSA (revised) on every measure in the table above; on the main session the two differ by up to 0.02 in accuracy and 0.06 in run-to-run agreement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.