acceptodds
Under review as a conference paper at ICLR 2027

Matching Only Readouts Tells Incomplete Stories: Temporal Context Shapes Decoding Gains in an Invasive BCI Foundation Model

Abstract

Frozen neural foundation models are often evaluated with readouts, which are small networks trained to predict behavior from a temporal window of neural signals or frozen features. The backbone can integrate activity outside the raw-signal readout's window, so matching readout architectures does not ensure the same temporal context. We study how much of the frozen-feature decoding gain depends on the readout's temporal context. We use a model pretrained by masked reconstruction on intracortical spikes and surface electrocorticography (ECoG) without behavioral supervision. On MC-RTT and MC-Maze, two datasets from macaque reaching tasks, we test whether additional temporal context reduces the decoding gain from frozen features. Each readout's input window is widened at fixed parameter count, while both receive inputs matched in time and the same explicit information about each prediction's position within the trial. We also examine whether Gaussian smoothing, which averages neighboring spike counts before decoding, reduces the gap without pretraining, selecting the kernel width on validation data. Widening reduces the gap between frozen-feature and raw-spike readouts by 89% on MC-RTT and 68% on MC-Maze, mainly through improved raw-spike decoding. At the widest window tested, the gap on MC-Maze averages 0.05 and is positive in every replicate. At that window, readouts using Gaussian-smoothed spikes perform nearly as well as those using frozen features on MC-Maze and better on MC-RTT. The results support temporal integration as a substantial source of the model's decoding gain on both tasks and leave open whether its frozen features also encode more abstract behavioral structure.

Primary area:
applications to neuroscience & cognitive science
Keywords:
neuroscience, neural decoding, neural foundation models, signal processing, representation analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UushOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the performance advantage of the pretrained BCIM in neural decoding tasks. Comparing readouts with different temporal window sizes and local smoothing shows that simply widening the readout window for raw neural signals yields R2R^2 values close to those of the frozen BCIM model on MC-RTT and MC-Maze datasets. This indicates that the decoding gains of frozen features primarily stem from temporal integration rather than encoding abstract behavioral structures.

Critical strengths
    1. The research question is clearly defined as the relationship between frozen-feature’s decoding gains and temporal context.
    1. The manuscript provides detailed reporting of experimental setups, including dataset descriptions, network architecture, readout protocols, and conducts comprehensive ablation studies.
Critical weaknesses
    1. The claim that frozen-feature gains stem from temporal integration is drawn from MC-RTT and MC-Maze datasets with similar data formats, prediction targets, and evaluation metrics. The key advantage of a pretrained BCIM is expected to lie in its generalization across diverse data, tasks, and subjects. However, the decoding tasks employed here may be solvable with simple temporal smoothing, thereby limiting the potential benefits in other scenarios.
    1. Motor decoding from macaque neural signals is inherently tied to temporal dynamics; hand velocity does not change abruptly but is strongly autocorrelated with past states. Hence, it is unsurprising that enlarging the readout’s temporal context substantially improves performance. Furthermore, the fact that simple Gaussian smoothing achieves competitive performance suggests that the chosen tasks may not sufficiently challenge the model’s representational capacity, thereby limiting the demonstration of a large-scale pretrained network's advantages.
Questions for authors
    1. Could the similar performance between frozen features and locally smoothed readouts be due to the simplicity of the current task setup, where a simple network with adequate temporal context suffices, making it difficult to demonstrate the value of the pretrained BCIM?
    1. Has consideration been given to comparing frozen features with other readout protocols under more challenging and generalization scenarios, such as cross tasks, cross species, or few shot learning, rather than only predicting hand velocity? These are typically regarded as the strengths of pretrained models.
    1. Table 1 shows that frozen features are more robust to randomized context than raw inputs. Does this indicate that their advantage lies in robustness and generalization, rather than merely predictive accuracy in a single scenario?
    1. Even with an enlarged temporal window, frozen features maintain a stable advantage over raw inputs (Figure 1 and Table 2). How can the remaining advantage be attributed solely to temporal integration?
    1. Since the manuscript claims that frozen-feature’s decoding gains may be unrelated to whether it encodes abstract behavioral structures, why not compare the neural manifolds generated by frozen features with those from other readout protocols, such as the linear model in Figure 2b? Such a comparison could provide more direct evidence.
2 comments
AuthorsOpenReview ↗

Reply to Reviewer Uush

Thank you for the review. Tables A and B are in the general response.

W1, W2 and Q1. The tasks may be solvable with temporal smoothing. We agree that these decoding tasks "may be solvable with simple temporal smoothing", which is the central point of the paper. MC-RTT and MC-Maze were chosen because they are standard benchmarks on which frozen-feature gains are reported. In our window comparisons, widening both readout windows removes most of the mean advantage of frozen features. This is a statement about what a comparison on these benchmarks can show, rather than the value of pretraining in other settings.

Q2. Harder settings: across tasks, across species, few-shot. The main comparisons concern velocity decoding, but two results bear on the question. Appendix C varies the training set from 25 to 100 percent of the training split. Since submission, we also evaluated an NDT2 checkpoint pretrained on human recordings and tested it on both macaque tasks. Across readout windows, its largest mean difference from causal Gaussian smoothing is +0.0037 [-0.0135, +0.0203] on MC-RTT and -0.0003 [-0.0032, +0.0028] on MC-Maze.

Q3. Does Table 1 indicate robustness and generalization? The control tests a narrower property. First, the backbone processes the intact trial. Neighbouring tokens in the readout window are then replaced with tokens from other positions in the same trial, with both readouts refitted. The smaller drop for frozen features is consistent with the backbone integrating temporal context before the readout. Robustness across tasks or sessions would need a different experiment.

Q4. Can the remaining advantage be attributed solely to temporal integration? We agree that these results do not establish temporal integration as the sole source of the remaining advantage. Section 4.1 states that the reductions "quantify changes in the decoding gap, not the causal contribution of temporal integration". At the widest window, the remaining difference has mixed signs across replicates on MC-RTT and stays positive in all five on MC-Maze.

Q5. Why not compare neural manifolds? We chose a functional comparison because the paper's question concerns decoding through the readout. Section 4.3 predicts the projected frozen features linearly from raw spikes and trial position under index alignment. Those predictions replace the projected features in the trained head, without retraining the head. A geometric comparison would complement this analysis.

Thank you for your response. However, several fundamental issues remain inadequately addressed. First, the paper’s core conclusion rests on two highly similar datasets, making it difficult to demonstrate the key advantages of a pretrained model. Second, the explanation for the residual advantage of frozen features is insufficient. Accordingly, I maintain my original rating.

Reviewer_wDdiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper investigates how temporal context contributes to the behavioral decoding advantage of frozen features from BCIM, an invasive BCI foundation model pretrained through masked reconstruction. On MC-RTT and MC-Maze, the authors widen the readout windows while controlling parameter count within each representation and matching temporal alignment and output-position information. The frozen-minus-raw decoding gap decreases by 89% and 68%, respectively, primarily through improved raw-spike decoding. The study also evaluates Gaussian smoothing, randomized context, and linear feature substitution into a fixed readout head. Notably, validation-selected Gaussian smoothing yields decoding performance close to frozen features on MC-Maze and better performance on MC-RTT at the widest tested window. These findings support temporal integration as an important source of BCIM’s decoding advantage and highlight the importance of temporal context and preprocessing when evaluating frozen neural representations.

Critical strengths
  1. Detailed reporting and extensive empirical controls. The paper provides detailed descriptions of its experimental protocols and quantitative results. The window sweeps are complemented by controls involving temporal alignment, output-position information, projection width, randomized context, and direct per-bin prediction. Together, these analyses provide substantial evidence for the observed reduction in the decoding gap and help readers assess its robustness and limitations.
  2. A striking result from a simple Gaussian baseline. The strong performance of Gaussian-smoothed spikes is particularly interesting. At the widest tested window, this simple preprocessing baseline approaches frozen-feature performance on MC-Maze and outperforms it on MC-RTT. This provides a useful practical lesson: temporal preprocessing can substantially affect the apparent benefit of pretrained representations and deserves careful consideration in baseline design.
Critical weaknesses

The additional scientific insight and its scope remain limited. The central empirical analyses concern one frozen backbone and two macaque tasks. Although the magnitude of the gap reduction and the strong Gaussian baseline are useful findings, improved raw-spike decoding with additional context is an expected possibility when the competing representation already incorporates full-trial information. The theoretical results largely draw on established estimation principles and do not directly establish BCIM’s computation. I therefore remain uncertain whether the study provides sufficient insight beyond a carefully documented evaluation of this particular model and protocol. Evidence from another backbone, or a stronger comparison under matched temporal support, would help establish the broader significance.

Questions for authors
  1. Scope of the contribution. Which conclusions are intended specifically as findings about BCIM, and which are intended as broader claims about neural foundation-model evaluation? Please clarify what evidence supports the broader claims, beyond the general expectation that a restricted raw-input baseline may improve with additional temporal context.
  2. Practical implications. Beyond reporting window sizes and temporal preprocessing, what concrete evaluation recommendations follow from the results? Please distinguish recommendations supported by the present offline experiments from possible implications for causal, real-time BCI decoding.
2 comments
AuthorsOpenReview ↗

Reply to Reviewer wDdi

Thank you for the review. Tables A and B are in the general response.

Q1. Which conclusions are about BCIM, and which are broader? Sections 4.1 to 4.3 present findings about BCIM from the window sweep, Gaussian baseline, randomized-context control, and substitution analysis. The broader statement rests on the two kinds of evidence the review suggests, which we have now run. Across three independently pretrained backbones, the gain over raw spikes is smaller at the widest window than at the narrowest in every replicate (Table A). With the same recorded bins for backbone and raw readout, frozen NDT2 features score below raw spikes in the mean in every cell (Table B). We limit the broader statement to these three backbones and two tasks.

W1. An improved raw baseline with more context "is an expected possibility". While the direction is expected, two results go beyond it. With the same recorded bins, the frozen features of NDT2 do not merely lose part of their gain but score below raw spikes in the mean (Table B). Two public causal backbones also score below a Gaussian filter in the mean at every evaluated window (Table A). The theory in the paper motivates these comparisons and does not identify the computation performed by the BCIM backbone. A future version will say so where the theory is introduced.

Q2. Concrete evaluation recommendations. The offline experiments support four.

  • Report the frozen-feature gain at more than one readout window.
  • Include a smoothing baseline whose width is selected on validation data under the same window.
  • Where possible, give the backbone and the raw readout the same recorded bins.
  • State which data the backbone and any session-specific fit have seen.

The pattern also holds in offline evaluations with causal backbones and causal smoothing. Latency was not measured.

Reviewer_wDdiOpenReview ↗

Reply to Authors

Thank you for the additional experiments and clarifications. I am raising my rating from 2 (Weak rejection) to 3 (Weak acceptance).

The paper offers an interesting and useful perspective on the interpretation of frozen-feature decoding gains: stronger temporal baselines can substantially change the apparent benefit of pretrained representations. The added NDT2/NDT3 evaluations and the matched-temporal-support analysis meaningfully address my concerns about reliance on a single backbone and unequal access to temporal context. Although the findings remain limited to the evaluated settings, the new evidence strengthens the paper’s contribution.

I ask the authors to address the following points in the revised manuscript:

  1. Release of the added experiments. Please clarify whether the public release will include the code, configurations, checkpoint identifiers, and per-replicate results for Tables A and B, in addition to the original experiments.
  2. Complete experimental documentation. Please incorporate the new results into the manuscript and document the feature-extraction procedures, readout training and model-selection rules, causal Gaussian implementation, accessible temporal support, and uncertainty estimates. Absolute decoding scores should accompany the reported differences. Please retain the distinction between causal offline evaluation and demonstrated streaming or real-time performance.
  3. Scope and future coverage. Please keep the conclusions bounded by the evaluated backbones and tasks. Broader coverage of independent recordings, tasks, backbones, and deployment settings would be valuable for establishing more general evaluation guidance. I regard this as an important direction for future work, rather than a request for additional rebuttal experiments.
Reviewer_4PN4OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper examines whether decoding improvements from a frozen invasive neural foundation model reflect its access to temporal context beyond that available to a raw-signal readout. Using BCIM, pretrained on intracortical spikes and surface ECoG, the authors show that widening readout windows substantially reduces the frozen-feature advantage on two macaque reaching datasets. Gaussian smoothing provides a competitive baseline. These results offer a useful methodological caution, but I find the scientific objectives insufficiently resolved, the rationale for multimodal training inadequately tested, and the evidence too narrow to establish a substantial general contribution.

Critical strengths

The paper addresses a legitimate evaluation concern: matching readout architectures does not necessarily match the information available to those readouts. The window sweeps, validation-selected smoothing baselines, and controls for temporal alignment and output position are useful. I also appreciate the authors’ acknowledgment that reduced decoding gaps do not quantify the causal contribution of temporal integration and that their analyses do not establish whether the model learns abstract behavioral representations.

Critical weaknesses
  1. The goals and criteria for success need sharper definition.

    The introduction states an operational question about temporal context, but the broader scientific objective remains unclear. Is the paper evaluating a new foundation model, explaining what reconstruction pretraining learns, establishing a general benchmark confound, or testing hypotheses about neural computation? These objectives require different evidence.

    In particular, “abstract behavioral structure” is not defined sufficiently to support a discriminating test. Improved decoding, temporal integration, denoising, and latent-state estimation are not mutually exclusive explanations. The paper needs explicit hypotheses, predicted outcomes, and criteria for distinguishing these possibilities. As written, the analyses demonstrate sensitivity to evaluation choices more convincingly than they explain the learned representation.

  2. The rationale for combining ECoG and spikes is insufficiently justified.

    These recording modalities have substantially different relationships to underlying neural activity. Spike counts measure local action-potential events, whereas surface ECoG measures aggregate extracellular field activity with different spatial sampling and signal characteristics. Recording geometry matters to the interpretation of these measurements; experimental comparisons directly demonstrate differences in the spatial sampling of ECoG, local field potentials, and multiunit activity. Dubey and Ray, 2019

    The authors accommodate modality differences computationally through separate tokenizers, an initial modality-specific block, and different reconstruction losses. This explains how both data types can enter one architecture, but does not establish why sharing the subsequent representation is beneficial or scientifically meaningful. The manuscript needs a clear account of what is expected to transfer across these measurement processes.

    A spike-only pretraining control, compared with joint pretraining under matched training resources, is particularly important. Downstream surface-ECoG decoding remains untested, and the descriptive sEEG analysis does not establish cross-modal transfer. Consequently, the contribution of ECoG pretraining to the main spike-decoding results remains unknown. An engineering objective based on predictive performance is legitimate, but it should be stated explicitly and evaluated accordingly; shared network weights alone do not establish shared biological mechanisms.

  3. The controls do not isolate the proposed explanation sufficiently.

    The manuscript includes numerous controls, but the central comparison still gives the backbone full-trial, noncausal access while constraining the raw readout to a local window. Widening the latter improves its performance, which is informative, but does not identify what computation the backbone performs. Competitive Gaussian smoothing likewise establishes the strength of a baseline without demonstrating that the frozen representation implements an equivalent operation.

    A more decisive experiment would independently manipulate the backbone’s temporal access and the readout’s window, with explicit matching of accessible recording times. A causal or fixed-latency evaluation would also clarify the relevance to online BCI use. Raw and frozen readouts retain different parameter counts and input dimensions, so the existing within-representation parameter controls do not fully resolve differences in readout capacity.

  4. Dataset provenance and independent generalization require stronger treatment.

    The principal window experiments use one recording per dataset. Repeated splits and optimization seeds quantify variability conditional on those recordings, rather than replication across subjects or sessions. Furthermore, Appendix D states that session-level overlap between pretraining and several downstream datasets has not been verified. This does not invalidate the paired window intervention, but it prevents a clear interpretation of the frozen model’s performance as generalization to unseen recordings.

    The main text would benefit from a consolidated dataset table specifying subjects, sessions, recording locations, signal types, preprocessing, trial construction, prediction targets, and pretraining overlap. An audited evaluation on sessions excluded from pretraining is needed.

  5. The presentation obscures a relatively narrow empirical conclusion.

    The evidence is distributed across several protocols with different alignment rules, projection widths, partitions, and readout settings. Extensive appendix material documents these distinctions, but makes it difficult to connect each experiment to a specific hypothesis. The theoretical results motivate temporal filtering without identifying the computation of the trained backbone, and the empirical analysis concerns only one pretrained model.

    The paper would be stronger with a clearer sequence of hypotheses, datasets, decisive controls, and conclusions. Replication on another independently pretrained model would help establish whether the observation extends beyond this checkpoint.

Questions for authors

How can models bridge ECoG and spikes across orders of magnitude in spatial resolution or more? An introduction with hypothesis, questions, and goal would help readability. I understand that saying BCI can be trendy. But what sort of BCI application is this work trying to address?

1 comment
AuthorsOpenReview ↗

Reply to Reviewer 4PN4

Thank you for the detailed review. Below we address the five weaknesses in order, followed by the questions, referring to Tables A and B in the general response.

W1. "The goals and criteria for success need sharper definition." The paper asks one operational question, namely how much of the decoding gain from frozen features depends on the temporal context given to the readout. We chose an operational question over a hypothesis about neural computation because the evidence consists of decoding comparisons. These comparisons show how a measured gain changes with readout context. They do not identify what the backbone computes or separate temporal integration, denoising, and latent-state estimation. The predicted outcome is that a gain coming from context outside the readout window shrinks when both readout windows are widened. In a future version, the first paragraph will state the question, this prediction, and the criterion.

W2 and Q1. "The rationale for combining ECoG and spikes is insufficiently justified." Joint pretraining is an engineering choice that provides one architecture for several invasive modalities, and it combines modality-specific input layers with shared backbone blocks. The motor decoding comparisons use spikes. We leave open whether anything biological transfers between the modalities and whether ECoG pretraining improves spike decoding. As the review suggests, a future version will state this engineering objective explicitly.

W3. "The controls do not isolate the proposed explanation sufficiently." Table B runs the experiment the review describes. It varies the temporal access of the backbone and the readout window separately, while giving the raw readout the same recorded bins. Frozen NDT2 features score below raw spikes in the mean in all eight cells. Table A adds two causal backbones, each compared against a causal Gaussian filter. We read the Gaussian result as a comparison with a strong baseline, not as evidence that the backbone implements the same operation. On readout capacity, the primary sweep holds the parameter count fixed within each representation. Raw and frozen readouts still have different parameter counts. Appendix C repeats the window comparison at a second projection width.

W4. "Dataset provenance and independent generalization require stronger treatment." We checked the BCIM pretraining corpus against the public archives. Neither evaluated MC session is in the corpus, whereas other sessions of both animals and all twelve evaluated Perich sessions are. Appendix D named indy_20160627_01 as the MC-RTT source session, but the evaluated session is indy_20170202_02. That session is absent from the cited Zenodo release. Neither MC session is in the pretraining data of NDT2 or NDT3. As a result, the MC results in Table A come from sessions outside the pretraining data of all three backbones. In a future version, the main text will carry a dataset table with the fields the review lists, including pretraining overlap.

W5. "The presentation obscures a relatively narrow empirical conclusion." A future version will follow a single sequence from question and datasets to controls and conclusions. Tables A and B provide the replication on independently pretrained models that the review requests.

Q3. Which BCI application? The setting is offline decoding of hand velocity from intracortical spikes, where frozen-feature gains are reported on these benchmarks. Comparisons with NDT2 and NDT3 use only current and past bins.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful reading. We also thank them for recognising the evaluation concern as legitimate (4PN4), the validation-selected Gaussian baseline as a practical lesson (wDdi), and the research question as clearly defined (Uush). Both experiments requested across the reviews (4PN4, wDdi) have been run. These add evidence from another pretrained backbone and compare the backbone and the raw readout on the same temporal support. All results are offline, on MC-RTT and MC-Maze, with five split-and-seed replicates of one recording per task.

1. Two more pretrained backbones. NDT2 and NDT3 are public causal models pretrained independently of BCIM. Neither evaluated session is in their pretraining data, and no backbone is fine-tuned on the evaluated sessions. Table A reports the narrowest and widest readout windows. For BCIM these are 100 ms and 900 ms, symmetric around the token nearest the decoded time. For NDT2 and NDT3 they are 20 ms and 520 ms of current and past bins, with a causal Gaussian filter.

Table A. Frozen features against raw spikes and against Gaussian smoothing on MC tasks

Each cell reports the mean paired difference in R2R^2 over five replicates and the number of positive replicates.

Backbone, outputTaskFrozen minus raw, narrowestFrozen minus raw, widestFrozen minus Gaussian, narrowestFrozen minus Gaussian, widest
BCIM (submitted)MC-RTT+0.181, 5/5+0.019, 3/5Table 2 of the submissionTable 2 of the submission
BCIM (submitted)MC-Maze+0.162, 5/5+0.052, 5/5Table 2 of the submissionTable 2 of the submission
NDT2 (added)MC-RTT+0.338, 5/5-0.037, 0/5-0.066, 0/5-0.037, 0/5
NDT2 (added)MC-Maze+0.390, 5/5-0.006, 0/5-0.050, 0/5-0.007, 0/5
NDT3, covariate-token outputs (added)MC-RTT+0.296, 5/5-0.140, 0/5-0.104, 0/5-0.138, 0/5
NDT3, covariate-token outputs (added)MC-Maze+0.415, 5/5-0.044, 0/5-0.025, 0/5-0.047, 0/5
NDT3, spike-token outputs (added)MC-RTT+0.367, 5/5-0.075, 0/5-0.033, 0/5-0.073, 0/5
NDT3, spike-token outputs (added)MC-Maze+0.438, 5/5-0.016, 0/5-0.001, 2/5-0.019, 0/5
  • Gain over raw spikes. For every backbone on both tasks, the gain is smaller at the widest window than at the narrowest in every replicate. For NDT2 and NDT3, the frozen features score below raw spikes at the widest window in every replicate.
  • Gaussian smoothing. NDT2 and both NDT3 outputs score below causal Gaussian smoothing in the mean at every evaluated window on both tasks.

2. The same recorded bins for backbone and raw readout. Table B restricts the NDT2 input to the LL most recent bins and reads its features over the current bin and KK past bins. The raw readout receives the same available recorded bins, up to L + K bins.

Table B. NDT2 with restricted backbone input crossed with the readout window

Each cell reports the mean paired difference in R2R^2 (frozen minus raw) over five replicates, the percentile bootstrap interval, and the number of positive replicates.

LKRaw windowMC-RTTMC-Maze
101 bin-0.022 [-0.031, -0.015], 0/5-0.047 [-0.052, -0.042], 0/5
11011 bins-0.049 [-0.056, -0.040], 0/5-0.022 [-0.026, -0.017], 0/5
15015 bins-0.088 [-0.105, -0.069], 0/5-0.076 [-0.082, -0.071], 0/5
151025 bins-0.043 [-0.060, -0.024], 0/5-0.008 [-0.012, -0.003], 1/5

Frozen NDT2 features score below raw spikes in the mean in all eight cells. We also tested K=0K = 0 with LL in {1, 5, 10, 15, 25}. NDT2 and both NDT3 outputs score below raw spikes on the same bins in every replicate on both tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.