acceptodds
Under review as a conference paper at ICLR 2027

Compact Spatiotemporal Representations via Summary Tokens for Human Intracranial Recordings

Abstract

Intracranial EEG (iEEG) provides high-resolution measurements of human brain activity, but building models that transfer across subjects and tasks is complicated by irregular electrode placement and by downstream tasks that rely on information at different spatial scales. Existing iEEG foundation models typically preserve channel-level structure during pretraining while leaving cross-channel aggregation to downstream adaptation, where labeled data are limited. We introduce a factorized spatiotemporal masked autoencoder that preserves channel-level tokens while learning two complementary summary representations: temporal summary tokens that aggregate information within each channel and spatial summary tokens that aggregate information across channels. During pretraining, an information bottleneck enforces reconstruction through these summary tokens, explicitly encouraging them to capture temporal and population-level structure. For downstream decoding, we use the learned summary representations across a range of iEEG tasks. Our results show that explicitly training the summary representations during pretraining improves downstream transfer across tasks with different spatial requirements. Further, we show that these representations capture task-relevant information and selectively aggregate signals from channels that contribute to downstream decoding.

Primary area:
applications to neuroscience & cognitive science
Keywords:
iEEG, neural decoding, masked autoencoding, neuro-AI

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_KvmROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method, MATCHA. It learns per-channel temporal and cross-channel spatial summary tokens through a masked autoencoder whose reconstruction predictor sees summaries but not visible content tokens. Brain Treebank pretraining transfers to speech, memory, behavior, and interictal spike datasets.

Critical strengths
  • The reconstruction bottleneck directly tests whether pretraining learns useful population summaries rather than leaving aggregation to downstream labels.
  • Du-IN subject-level results and task-related neural analyses support the design.
Critical weaknesses

(1) The evaluation of speech decoding is potentially problematic. The authors state that they use subject-level normalization followed by random training, validation, and test splits. This raises a significant concern: the normalization process uses the test data, which causes data leakage. The proper way should be to split the data into training and test sets, and the normalization parameters should be estimated from the training set and applied to both the training and test sets. The original Du-IN dataset used channels selected according to their decoding contribution. This creates another potential source of inflated performance. (2) The contribution of the information bottleneck is not fully isolated. The proposed method combines several design choices: factorized spatiotemporal attention, dedicated spatial and temporal summary tokens, an information bottleneck, and downstream attention pooling. It remains unclear whether the improvement is primarily due to the proposed bottleneck principle or to the particular number, placement, and granularity of summary tokens. (3) The evidence of generalization across heterogeneous intracranial recordings is limited. It is well-known that intracranial recordings vary considerably due to electrode placement, anatomical coverage, recording hardware, referencing, and underlying pathology. The manuscript does not convincingly establish that the learned spatial summaries generalize robustly across these sources of heterogeneity, as the pretraining dataset, Brain Treebank (ten pediatric epilepsy patients), is too small and clinically specific. (4) The physiological interpretation of summary-token attention is limited. The authors visualize spatial and temporal attention patterns and report consistency with known speech-related regions and epileptiform activity. However, attention weights should not automatically be interpreted as the importance of specific neural populations or anatomical regions. It may just reflect correlated signals, electrode coverage, and statistical features associated with the task rather than independent physiological contributions.

Questions for authors
  1. Can the authors clarify whether repeated word trials from the same recording sessions appear in different partitions?
  2. Was channel selection performed independently of the test data, or were the ten channels per subject inherited from the original Du-IN study?
  3. Do the reported results represent within-subject, within-session decoding, or generalization across independent sessions? How would performance change under a session-disjoint evaluation protocol?
  4. Could the authors clarify whether the bottleneck and non-bottleneck models have identical encoder architectures, summary-token configurations, pretraining objectives, and computational budgets, differing only in the decoder's access to content tokens?
  5. How sensitive are the results to the number and granularity of spatial and temporal summary tokens?
  6. How robust are the learned representations when transferred to subjects with substantially different electrode configurations or anatomical coverage?
  7. How does performance change when anatomical location information is missing or when the downstream electrode coverage differs considerably from the pretraining distribution?
  8. Could the authors compare the proposed summary-token representations against simpler aggregation strategies, such as mean pooling, attention pooling over channel tokens, or a standard CLS-token representation, under matched model capacity and pretraining conditions?
Reviewer_ky1uOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MATCHA, a factorized spatiotemporal masked autoencoder as an iEEG foundational models. MATCHA introduces learnable spatial and temporal summary tokens during the model's pretraining phase. Within the MAE architecture, only these summary tokens are used to reconstruct masked patches. The learned summary representations are then used for downstream finetuning.

Critical strengths
  1. This paper introduces temporal summary tokens organized by channels and spatial summary tokens organized by time positions to reconstruct masked patches.
  2. Experiments cover various behavior, cognitive, and clinically relevant decoding tasks, achieving competitive results on most evaluation datasets.
Critical weaknesses
  1. The writing of this paper needs to be improved, and the motivation needs to be more clearly clarified. The explanation of the time aggregation requirement is weak, and the authors does not clearly explain why separate spatial and temporal summary tokens need to be constructed.
  2. The paper takes "limited downstream task annotation data" as an important motivation, arguing that learning spatio-temporal aggregation during the self-supervised pre-training stage can alleviate the burden of downstream learning. However, the current experiments mainly compare the decoding performance under a fixed annotation quantity, and does not directly verified whether this method improves label efficiency.
  3. In the ablation experiments, only comparisons were made between the presence vs absence of the information bottleneck and different downstream summary readouts. However, sensitivity analysis regarding the core design-related hyperparameters is lacked. The number of temporal/spatial summary tokens directly determines the bottleneck capacity.
  4. Minor comments:
  • The boldface highlighting for ds005523 in Table 1 is incorrect.
  • The caption of Figure 3 is problematic: the model does not outperform the comparison models on all subjects.
  • Finetuning involves an important design component of this paper, so I believe this part could be described in the main text.
Questions for authors

Please see weaknesses.

Reviewer_4nXeOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper argues that existing iEEG FMs face a bottleneck, which is that spatial and temporal aggregation is often left to downstream finetuning, where labeled data is limited. To address this problem, the authors propose MATCHA combining temporal and spatial summary tokens with an information bottleneck for masked reconstruction. The model was pretrained on 29 hours of iEEG and investigates how the summary tokens learn spatial and temporal information. The model is evaluated on 8 iEEG datasets to show the improvements over previous iEEG foundation models.

Critical strengths
  1. This paper addresses a very relevant research gap, especially the lack of explicit spatial and temporal aggregation during pretraining for iEEG FMs.

  2. The authors are careful with the experimental design, using cross-dataset evaluation and direct ablation studies. Fig. 3 also shows that their design choices lead to real improvements, especially the information bottleneck (0.532 to 0.638 macro-F1).

  3. The experiments show that learning summary representations during pretraining improves downstream transfer across different iEEG tasks.

  4. The authors also observe that the summary tokens capture task-relevant spatial and temporal information, especially the brain regions related to speech and temporal patterns of epileptiform activity.

  5. I also find the information bottleneck design quite interesting. This force the reconstruction to go through summary tokens during pretraining. I think this is a simple but meaningful design choice, which could provide useful insights for future iEEG representation learning.

Critical weaknesses
  1. The authors should provide more ablation studies to show to what extent the proposed information bottleneck actually helps. For example, Tab. 10 shows that the improvements are not consistent across all datasets, especially AJILE12 and ds004752.

  2. I suggest the authors should investigate the method with more pretraining data. The current model is pretrained on only 29 hours of iEEG, while existing iEEG FMs have used much larger datasets. You should show whether the improvement can be maintained with larger and more heterogeneous pretraining data.

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.