acceptodds
Under review as a conference paper at ICLR 2027

Neural Data Needs Semantic Tokenization: Behavioral Events as Session-Transfer Token Boundaries

Abstract

Extracellular electrophysiology records a different set of neurons in every session. Neural foundation models embed each neuron and each session into their tokens, so every new session is an input they have never seen, and they fail to generalize to it. A tokenizer for new sessions needs a unit that every session shares and that carries behavior. Population activity offers such a unit. It evolves on a low-dimensional manifold that persists across neuronal turnover and across animals once sessions are aligned. This manifold changes regime at task events such as stimulus onset and movement onset. Within each regime the population occupies a state, the part of the manifold it spans between two events, and each state carries its own behavioral meaning. We propose Tokenization with States (TWS), which segments each trial, one repetition of the task, at these events and converts every state into tokens of population geometry, with no neuron or session embedding. On held-out International Brain Laboratory (IBL) sessions, TWS decodes movement even from regime boundaries that carry no information about the target, while a per-neuron foundation model pretrained on those sessions decodes at chance. Frozen after training on mice alone, TWS transfers to macaques and Utah arrays with only a linear probe. On reach direction, a target that its boundaries do not define, it achieves a Matthews correlation of 0.23, where the event time alone achieves 0.01. For cross-session generalization, the token matters more than the model on top.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Neural Foundation Models, Neural Tokenization, Cross-Session Generalization, Neural Manifolds, Population Representations, Electrophysiology

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UzKqOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Authors propose a new model architecture that uses trial structure to define a set of cross-attention pooling tokens which aggregate information across neuronal population activity across structured time bins of the trial.

Critical strengths
  • The problem formulation is an important direction in neuroscience as prior models usually rely on labeled data collection and fine-tuning to achieve behavior decoding.
  • Authors perform extensive ablation on the different modeling components and highlight which design choice matter more than others.
Critical weaknesses
  • I have significant concerns about the write-up. I think the write-up requires a significant rewrite and it doesn't meet the high bar of ICLR right now. I found many sentences very hard to read and structurally ill-posed.
  • The manuscript fails to describe the details of the proposed methodology in the main text. The model formulation is just limited to one algorithm table, where the definitions of the learnable parameters are dumped into the Appendix. I understand that they are limited to 9 pages, but I think they should still be able to describe their methodology within the limits.
  • Even though I am familiar with the domain and can understand what the authors try to convey in the related work, I dont think it does a good job on explaining the prior literature, especially for someone who is not very familiar with this domain. they should explain the relation of each work to the proposed problem more clearly.
  • Can they explain what they mean by "a unit with two properties: (1) a single decoder can use it in every session, and (2) it carries behavioral meaning. We look for such a unit in population activity, the joint activity of the recorded neurons". Similar to SPINT, or POYO, their model also relies on cross-attention across neuronal population. In their model formulation, are they using multi-unit spikes rather than sorted units? if so, I'd like to see the proposed model's performance with sorted spikes where the neuron identity changes from one session to another. I wonder if the cross-attention module will be able to transfer its knowledge to a new session without fine-tuning.
  • In line with my previous comment, what modeling component here makes their cross-attention module work but other models' fail?
  • In terms of methodology, my primary concern is that in Table 2, it looks like the model heavily relies on the state boundaries. These are unknown in causal and real-time settings, and using this information for inference significantly hinders models applicability for real-world neuroscience settings.
  • In L154-155, they say: "This step uses only the unlabeled activity of that session", but aren't they using b_stim and b_move, which are supervised variables requiring task knowledge? Also, in the abstract, they also say "A tokenizer for new sessions needs a unit that every session shares and that carries behavior", which is not necessarily accurate when fine-tuning is allowed.
  • The baseline results are also concerning, I cannot see any setting where POYO and CEBRA achieves reasonable performance except for table 17 where the authors provide the event boundaries. This makes me think that if something in their preprocessing and/or normalization is the reason why the baseline models fail.
Questions for authors

Please see weaknesses above.

Reviewer_nskKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes TWS, a tokenizer for multi-session electrophysiology that avoids neuron and session embeddings. Each trial is split into three windows at recorded task events (stimulus onset, first movement) with learned offsets; per-neuron mean rates within each window are encoded by a shared MLP and attention-pooled with two learned queries per window. A decoder is then attached for probing, and the model is tested in cross-dataset transfer, leave a session out validation, and ablations. The paper asserts that the tokenizer, not the backbone, determines cross-session decoding, per-neuron models collapse on unseen sessions, and a mouse-trained TWS transfers across species.

Critical strengths

As far as I can tell, their Gallego reach direction result is zero shot reach direction classification, beating the other models in the transfer setting (including the matched identity-free PerceiverIO, Table 20). There does seem to be signal that the method has learned something that transfers.

Critical weaknesses

My first concern is that the headline target, ‘Movement,’ is nonstandard and unrepresentative, and therefore, it fails to support their claims about relative model performance (which supports their claim about tokenization being what matters most). Prior IBL benchmarks (NEDS, POYO+, NDT2) decode choice, block, wheel velocity and whisker ME; on these TWS is at chance cross-session (Table 12) and the paper introduces a Type A/B idea to account for it. The target the paper leads with instead is a binary derived from the same first-movement time that the tokenizer receives as input, and an onset-only probe scores 0.908 against 0.565 for TWS (Table 2). When every representation is given the onset, all models are statistically identical (Tables 17, 27). The authors anticipate this with Table 2b where they trained and evaluated with fixed, random, or shuffled boundaries. In this case, TWS retains 0.17–0.22 MCC while the onset probe scores 0. I take this as the paper’s real estimate of neural movement decoding on IBL. But PerceiverIO, which receives no boundary at all, scores 0.303 (Table 1), so without the onset input the identity-free baseline is ahead. 0.17–0.22, not 0.565, is the neural decoding number for IBL claims and it reverses the TWS–PerceiverIO ordering of Table 1.

Repeated throughout is the claim that the model “fits nothing per session” (zero-shot!) and that “our cross-session probe fits nothing to the test session.” This is contradicted by the evaluation protocol: “Its unlabeled trials set the per-neuron z-score, and continuous targets are standardized within each session, which uses the label statistics of the test session” (Section 4.1). Per-unit mean and variance from the held-out session are fitted per-session quantities, and the RT standardization uses held-out labels. If you did not do this (e.g. per-trial normalization only, RT standardized with training-session statistics), would the model still transfer? I would say this also mandates a SPINT comparison, since SPINT’s few-shot unlabeled calibration uses the same kind of test-session statistics.

Questions for authors

Why did the authors choose to not compare to SPINT?

Why did the authors choose 6 Grassmanian tokens? Why not 7?

Appendix G shows kdim ∈ {1, 2, 4} are within 0.003 and removing Gram–Schmidt changes nothing (G.2); Table 3 shows the backbone barely matters. What does the Grassmannian/subspace framing contribute beyond attention pooling over neurons with 2 queries per window? If the geometry is not important, the method should be described as a set encoder on event-aligned mean rates.

The model-free “Pattern” summary (Table 19) matches TWS on 4 of 5 transfer targets. What does the learned encoder add beyond reach direction?

Gallego and NLB have continuous kinematics, the standard targets for those datasets. Can TWS decode them?

Reviewer_whRfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes Tokenization with States (TWS) for transferring neural representations across recording sessions with different recorded neurons. TWS divides trials at behavioral events and aggregates population activity within each segment into tokens without neuron or session embeddings. The authors evaluate behavioral decoding on held-out IBL sessions and transfer a frozen mouse-trained encoder to additional mouse and macaque datasets using linear probes. Validation includes comparisons across tokenizers and backbones, boundary perturbations, and controls using event timing alone. The strongest evidence of transfer beyond information supplied by the boundaries is macaque reach-direction decoding, where TWS achieves an MCC of 0.23 versus 0.01 for event timing alone.

Critical strengths
  1. TWS addresses a practical obstacle to transferring neural models: each session records a different population of neurons. Its permutation-invariant aggregation produces a fixed set of tokens without learning new neuron or session embeddings. The design is relatively simple, and the backbone comparisons suggest it can work with several architectures.
  2. The macaque reach-direction experiment provides meaningful evidence of transfer beyond the information supplied by behavioral boundaries. With a frozen mouse-trained encoder and a linear probe, TWS achieves an MCC of 0.23, compared with 0.01 from event timing alone and 0.13 from an untrained encoder. This supports the usefulness of the learned representation across species and recording technologies.
Critical weaknesses

The exps do not separate the benefit of behavioral-state segmentation from the benefit of supplying behavioral information. In the tokenizer comparison (4.4, Table 3), TWS receives first-movement times, whereas PerceiverIO does not. The same event is used to define movement occurrence and is closely related to reaction time. Consequently, the movement MCC difference of 0.565 versus 0.303 changes both the tokenizer and the available information. The authors’ controls confirm the importance of this distinction: timing alone achieves 0.908 MCC, and supplying the boundary to both models leaves differences of only +0.001 MCC and −0.001 reaction-time R2, neither statistically significant (Tables 2 and 27). What remains unresolved is whether organizing neural activity into behavioral states improves transfer over another identity-free encoder given the same event information, on a target not determined by those events. The reach-direction result is encouraging: TWS achieves 0.232 versus PerceiverIO’s 0.197. However, PerceiverIO leads on Steinmetz choice and NLB movement (Table 20), so successful transfer alone does not establish the benefit of semantic segmentation. This matters because the paper’s main methodological claim concerns the choice of neural tokens, while its largest reported gains come from a comparison that also changes access to behavioral information.

Questions for authors
  1. How much of the macaque reach-direction result depends on aligning tokens to behavioral events? Can the fixed- and shuffled-boundary models from Table 2 be evaluated under the frozen-transfer protocol of Table 20, applying each boundary rule consistently during training and transfer? Since timing alone does not predict reach direction, this would test whether the 0.232 MCC specifically benefits from behavioral alignment rather than population aggregation alone.
  2. Does TWS’s reach-direction advantage over PerceiverIO (0.232 versus 0.197) persist when both encoders receive the same event information during training and encoding? Tables 17 and 27 append timing to the final probe. Matching the information available to the encoders would address the separate question of whether organizing activity into state tokens improves the representation.
  3. Which POYO features and pooling operation are used for the probe, and can those same features decode movement using a separate probe within each session? Table 12 reports movement MCC of 0.013 even when training and test trials share sessions, although the probe pools across sessions. Clarifying whether this reflects incompatible session coordinates or poor behavioral decoding within individual sessions would help assess the proposed explanation for POYO’s failure to transfer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.