acceptodds
Under review as a conference paper at ICLR 2027

R-NDT: Robust Neural Data Transformer for Generalized Causal Motor Decoding

19 views

Abstract

Intracortical brain motor decoding aims to interpret and translate neural spike activity into behaviors. Causal decoding models that can generalize across variations, such as sessions, behaviors, subjects, and recording sites, are critical for real-world brain-computer interface (BCI) applications. However, every recording session presents the decoder with a new set of neural units and a continuous stream of nonstationary population spike activity. In this work, we present R-NDT, a robust neural data transformer that is designed to adapt to new recording sessions without need to re-learn unit/session embeddings, and it decodes long-term recordings in a streaming causal manner. Specifically, 1) unit identity is inferred from a few seconds of unlabeled calibration activity by a permutation-equivariant encoder to weight units in a population read-in; 2) a sliding-window causal transformer processes spike count bins with a bounded key-value cache; 3) a moving-average state conditions every block through adaptive layer norm; and 4) behavior covariates read the latent state through cross-attention. We pretrained R-NDT using several public datasets. After pretraining, we evaluated R-NDT generalization by fine-tuning it on cross-session, cross-type, cross-subject, and cross-site downstream behavior decoding tasks. Our results show that, on average across 20 sessions, R-NDT outperforms the decoding performance of existing decoding models, which has implications for future BCI decoders.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain-Computer Interfaces, Neural Transformer, Real-time Motor Decoding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FgN2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes R-NDT for causal motor decoding across heterogeneous intracortical recording sessions. It derives unit representations from a short unlabeled calibration segment using a permutation-equivariant set encoder, then maps a variable-sized neural population into one token per time bin. A sliding-window causal Transformer with a bounded KV cache and multi-timescale drift memory processes the resulting stream, while a query-based readout predicts behavior. The model is pretrained on 156 public sessions through masked spike-count reconstruction and subsequently fine-tuned with labeled data from each target session. Evaluation covers 20 held-out public sessions under a per-trial causal protocol, as well as T-RT and NPCS recordings under a continuous whole-block protocol. The full model achieves strong aggregate performance against the included baselines, and its smaller from-scratch variant is also competitive.

Critical strengths

Strength 1

A promising architecture evaluated under a practically relevant streaming protocol. The combination of a flexible population read-in, bounded-memory causal attention, and a slower drift state addresses important constraints of continuous neural decoding. The whole-block evaluation avoids resetting the decoder at every short trial and is therefore particularly relevant to deployment. Across several datasets, R-NDT performs strongly against recent baselines. Notably, the smaller R-NDT-SS model remains competitive without pretraining, which supports the value of the architectural design itself.

Strength 2

A useful route for learning from unlabeled neural recordings. The masked-count objective allows R-NDT to pretrain across many sessions without aligned behavioral labels. The paper reports improved downstream performance as the amount of pretraining data increases. This is a promising direction for incorporating larger collections of neural recordings, potentially including recordings from other tasks for which suitable behavioral labels are unavailable. Such broader cross-task scaling remains a future possibility rather than a result established by this paper.

Critical weaknesses

Critical weaknesses

1. The case for unit-level inputs is insufficiently established. R-NDT relies on sorted units and their calibration statistics, but does not compare this choice with channel-level or latent-alignment approaches in performance and preprocessing cost. CRRL, RAT, and SABLE offer relevant alternative approaches to cross-session representation learning. Channel signals can also drift; my concern is that the specific benefit of the unit-level design remains unclear.

2. The contributions of key components are insufficiently isolated. The paper lacks controlled ablations for the CC-based unit encoder and drift memory, making it difficult to determine which components drive the gains. In addition, R-NDT-SS and R-NDT (FT) differ in both model size and pretraining, so their comparison does not isolate the effect of pretraining.

Questions for authors

Questions for authors

  1. Why use sorted units as inputs? What measurable benefit does unit-level calibration provide over channel-level or latent approaches? How do spike-sorting cost and errors affect the results?
  2. Could the authors provide size-matched controls and component ablations? In particular, a randomly initialized model of the same size, a learned embedding lookup in place of CC-based embeddings, and a model without drift memory would materially affect my assessment.
  3. Which parameters are updated during supervised fine-tuning on a target session? If the set encoder and W′W' also receive gradients from behavior labels, can the authors disentangle the benefit of unlabeled calibration from subsequent supervised updates?
  4. What is the practical value of pretraining? In Table 2, R-NDT (FT) exceeds R-NDT-SS by approximately 0.013 in mean R2R^2, with a much smaller difference on some datasets. Since these models also differ in size, what gain is attributable to pretraining at matched capacity? How does that gain compare with the pretraining compute cost?

I would be willing to raise my score if the authors provide controlled evidence clarifying the independent value of unit-level calibration and the sources of the performance gains.

Reviewer_RVpVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents R-NDT for causal decoding from intracortical recordings with different sets of neural units. The model pools each bin’s spike counts into a single fixed-dimensional token using unit weights inferred from a short calibration segment. These representations are processed by a sliding-window causal transformer with bounded attention and a moving-average memory conditioning the transformer layers. The model is pretrained using masked reconstruction of spike counts and then fully fine-tuned for decoding in motor tasks. The authors report improvements over several competitive decoding models.

Critical strengths
  1. Combining units into population-level tokens is a good engineering design. Inferring unit context from activity avoids fitting a separate embedding table for every session and allows the same model to handle different numbers and orderings of units. This is a useful design for heterogeneous neural recordings.

  2. The evaluation includes several competitive baselines and both trial-based and whole-block decoding. Both the pretrained and from-scratch R-NDT models outperform their corresponding POYO and POSSM models on average.

Critical weaknesses
  1. The contribution is limited. Architectural components, the overall pretraining paradigm, and the evaluation scope overlap with prior work. Pretraining followed by supervised subject-specific fine-tuning has already been investigated in NDT2 and NDT3 (Ye et al., 2025), and benefits from increasing pretraining scale have been shown. Mapping recording units into a fixed-dimensional latent space has been explored in SPINT, and the cross-attention decoder with behavioral queries resembles that of POYO. Whole-block decoding has also been evaluated in previous work, such as NDT3 and DPAD (Sani et al., 2024). While R-NDT combines these ideas in a structured way, it does not establish a clear methodological advance beyond their integration. The evaluation is also limited to velocity decoding in center-out and random-target reaching tasks. The contribution therefore mainly consists of improvements over competitive baselines.

  2. The potential for transfer without full fine-tuning is not evaluated. The authors emphasize that the architecture does not require subject-specific or session-specific parameters. This creates an interesting opportunity to test a pretrained model and decoder pair on a completely new subject or session, using the context estimated from 8 seconds of unlabeled activity. However, this opportunity is not evaluated in in the current settings, since the cross-session and cross-subject results show transfer only after fully fine-tuning the model, rather than decoding with a frozen model. A fully frozen cross-session or cross-subject test, or a frozen backbone evaluation with only a fitted decoder head would help establish how much of the learned representation transfers without updating the entire model. Relatedly, fine-tuning limited parts of the model could show which model components need adaptation for new sessions or subjects.

Questions for authors
  1. Can the authors provide an ablation of the moving average memory? This is one of the main architecture differences compared to baseline models and a central motivation of the paper, but its contribution is not isolated.

  2. How much nonstationarity or shift in unit distributions occurs in the whole-block recordings? Does the memory’s benefit vary with the severity of this shift or the neural calibration segment remain reliable in these settings?

  3. In Figure 3D, how many minutes and trials does the 10% setting correspond to for each dataset? Since sessions have different lengths, can the authors report the amount of labeled data in seconds or trials used to adapt to new sessions or subjects?

  4. R-NDT (FT) has more than 10 times as many parameters as R-NDT-SS. Does R-NDT-SS saturate at the current parameter count, or would a larger single-session model improve over it?

  5. How sensitive is decoding performance to the 8-second neural context duration and the time window selected for it?

Reviewer_yn2WOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces the Robust Neural Data Transformer (R-NDT) a model designed to adapt to new recording sessions without need to re-learn unit/session embeddings as well as decode longterm recordings in a streaming causal manner. It consists of several stages including inferring from unlabeled calibration activity, processing spike count bins with a sliding window transformer, maintaining a moving average state, and using behavior covariates to read the latent state through cross-attention. The experimental results show that R-NDT outperforms existing decoding baselines demonstrating improved cross-session generalization and streaming decoding performance.

Critical strengths
  • The main strength is the authors design of the novel architecture/pipeline which includes Inferring unit identity from spikes, Flexible unit size, Causal streaming, State across windows, and Label-free pretraining
  • The results demonstrate improved generalization performance across different recording days, subjects, datasets, and brain regions
Critical weaknesses
  • There is limited ablation analysis of each key component of the pipeline, neuron identity inference, cross-attention, etc.
  • There is a lack of discussion of the limitations of this method in terms of its sensitivity the calibration phase and the degree of drift over time
Questions for authors

Currently the model requires supervised fine-tuning on new sessions. How could this method be extended beyond offline decoding to real-world closed-loop BCI?

How much does each component of R-NDT contribute the the overall decoding performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.