acceptodds
Under review as a conference paper at ICLR 2027

iMINDBench: iEEG Multi-Institution Neural Decoding Benchmark

Abstract

Intracranial electroencephalography (iEEG) records electrical activity directly from the human brain, making it attractive for neural decoding. However, progress toward general-purpose iEEG foundation models is difficult to measure because fragmented datasets and processing pipelines obscure whether model gains persist across institutions, arise from modeling rather than preprocessing, or grow with broader supervision. Thus, we introduce iMINDBench, an iEEG Multi-Institution Neural Decoding Benchmark that aligns fifteen naturalistic decoding tasks across three independently collected institutional datasets to measure progress across heterogeneous recordings. We define standardized spectral and waveform preprocessing tracks and find that pretrained systems generally outperform non-pretrained baselines within their respective tracks, yet an engineered spectral baseline remains competitive with pretrained systems, especially on the two datasets with less downstream training data. The benchmark's harmonized targets and electrode metadata also enable a matched-task scaling study across subjects and institutions. We find that additional same-dataset subjects yield modest gains, whereas adding data from other institutions provides little aggregate benefit despite up to a 25-fold expansion of the supervised dataset. These findings highlight the need for general-purpose iEEG decoders that deliver consistent gains across institutions and make more effective use of cross-institution supervision.

Primary area:
datasets and benchmarks
Keywords:
iEEG, neural decoding, benchmark, naturalistic stimuli, multi-institution evaluation, preprocessing tracks, foundation models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xK3FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces iMINDBench, which extends Neuroprobe’s 15 naturalistic iEEG decoding tasks to two additional movie-watching datasets: Bang! You’re Dead from Cedars-Sinai and Pippi from Utrecht. It aligns stimulus labels and electrode metadata, provides fixed evaluation splits, and defines spectral and waveform preprocessing tracks, including documented model-specific variations. The experiments compare conventional decoders with pretrained models and examine whether adding labeled data from other subjects and institutions improves performance. The main findings are that preprocessing choices have a substantial effect, a simple Multi-STFT MLP also remains competitive with the pretrained waveform systems, and PopT-v2 gains relatively little from broader supervision.

Critical strengths
  1. The paper makes a convincing case for taking preprocessing seriously when comparing models, and this is something that previous benchmarks, in my opinion, might have skipped. In Section 4.2 the authors show that normalization alone can change baseline ROC-AUC by more than 0.10 in some settings. Removing the 0.5 Hz high-pass filter also reduces HTNet’s average performance from 0.612 to 0.589, with decreases across all three datasets. These changes can exceed the differences between models in Table 1. I appreciate that the paper distinguishes comparisons using the same inputs from comparisons between complete processing pipelines, and asks for compatible baseline reruns when preprocessing changes. This makes the benchmark more useful for understanding where improvements come from.

  2. Bringing these datasets into a common evaluation framework is a useful contribution. The datasets differ in their stimuli, languages, electrode coverage, and recording setups, so aligning their labels and electrode information requires substantial work. The shared task definitions, fixed splits, and reporting of preprocessing and pretraining data provide a practical basis for comparing models across institutions. If the code and supporting materials are released as described, this could make evaluation on multiple iEEG datasets considerably easier for other researchers.

Critical weaknesses
  1. Now, some of the conclusions that are made in the paper are quite a bit stronger than the evidence that currently supports them. Tables 1, 4, and 5 do not include confidence intervals or paired comparisons, making it difficult to judge whether small differences between models are reliable. The stability of the Main/Challenge grouping is also unclear, particularly for the shorter recordings, where relatively few validation examples may determine whether a unit passes the 0.60 threshold. The scaling analysis includes confidence intervals, but its cross-institution results are based solely on PopT-v2. This limitation is acknowledged in Section 7 and should also be reflected in the abstract and conclusion. The training setup needs clarification as well: Appendix I specifies 1,000 fine-tuning updates, but it is unclear whether this budget is used in every scaling setting and how often the target session is sampled as more data are added.

  2. Several protocol details still need clarification, despite the useful code release. It remains unclear whether the evaluated Neuroprobe subjects contributed other sessions to pretraining; this matters for interpreting generalization, although it would not automatically invalidate the stated evaluation. The supplied Pippi pipelines use 60 Hz notches, leaving open whether 50 Hz line noise was addressed upstream. The BYD/Pippi split builders also lack an explicit temporal gap, while some preprocessing uses surrounding signal from the full recording. Finally, some supplied defaults differ from the manuscript or earlier settings described in the README.

Questions for authors
  1. Were the evaluated Neuroprobe subjects excluded entirely from PopT-v2 and BaRISTA pretraining, or were other recording sessions from those subjects included? Please provide the subject/session lists for pretraining and evaluation, identify the checkpoints used, and clarify how any overlap affects the interpretation of Table 1.

  2. The supplied Pippi pipelines use 60 Hz notches. Were the source recordings already filtered at 50 Hz? If not, does applying the appropriate line-noise filtering change the Pippi results?

  3. Could you add confidence intervals and paired model comparisons to Tables 1, 4, and 5, accounting for observations from the same subject being related? Please also report validation class counts and show whether Main membership and model rankings are sensitive to the 0.60 threshold.

  4. Is the same 1,000-update fine-tuning budget used for within-session, within-dataset, and multi-dataset training? How are examples sampled in each setting, and how much exposure does the model receive to the target session as the pooled dataset grows?

  5. The supplied BYD/Pippi split builders do not appear to include a temporal exclusion gap, while the waveform preprocessing can load context across split boundaries. Was an additional safeguard used for the reported experiments, and are the results stable when a suitable gap is introduced?

10 comments
AuthorsOpenReview ↗

Rebuttal response 1/N

Thank you for your thoughtful review and for recognizing the importance of preprocessing-aware comparisons and a common evaluation framework across datasets. We respond to your concerns below with additional experiments, uncertainty analyses, and protocol clarifications. We welcome further discussion and suggestions for improving the work.

W1. some of the conclusions that are made in the paper are quite a bit stronger than the evidence that currently supports them.

Tables 1, 4, and 5 do not include confidence intervals or paired comparisons

We added participant-bootstrap confidence intervals and paired model comparisons; see our response to Q3 and Appendix L. We find that the confidence intervals support the existing patterns seen in the benchmarks leaderboards.

stability of the Main/Challenge grouping is also unclear

We agree that Main membership depends on the screening rule and validation support. Our Q3 response and Appendix M report a threshold sweep: Overall rankings within each input track are stable across the tested union thresholds.

cross-institution results are based solely on PopT-v2

We attempted to evaluate multi-dataset results with additional models, but found that significant changes to the proposed models were necessary to allow for competitive performance. The within-dataset and multi-dataset experiments require finetuning on multiple participants which means different electrode layouts must be mapped to the same single logit output. Our other evaluated model implementations use decoding heads tied to a fixed electrode layout; extending them to joint training across participants requires adapting and validating those heads. We attempted to replace the decoder heads with methods that allow for input channel variation during finetuning (tried: cls token, pooling), but found much poorer performance. More development is needed to get a clear signal from these additional models and baselines. We will clarify this in the abstract and conclusion.

budget is used in every scaling setting, how often the target session is sampled

We address these in our response to Q4.

W2. Several protocol details still need clarification, despite the useful code release.

Neuroprobe subjects contributed other sessions to pretraining

Other sessions from evaluated Neuroprobe participants were included in PopT-v2 pretraining. We added participant-exclusion results and recording/checkpoint provenance in Appendix N (Table 13); see the detailed response to Q1 below.

Pippi pipelines use 60 Hz notches, leaving open whether 50 Hz line noise was addressed upstream.

We will correct the released Pippi configuration to use 50 Hz harmonic notches; see Q2 below for the spectral checks and initial three-seed performance comparisons.

The BYD/Pippi split builders also lack an explicit temporal gap, while some preprocessing uses surrounding signal from the full recording

We follow Neuroprobe’s contiguous-block splitting convention, without an additional temporal exclusion gap. Preprocessing context can cross split boundaries, but are cropped to one-second target windows. We clarify this distinction in our response to Q5 below and will document it in the manuscript.

supplied defaults differ from the manuscript or earlier settings described in the README

We reviewed the supplied code against the manuscript and saved experiment configurations. We confirmed that the supplied Logistic and MLP tolerance defaults differ from those used for the reported Multi-STFT experiments. We will provide explicit reproduction configurations matching the saved settings and clarify their use in the README. We welcome any additional code–manuscript discrepancies the reviewer could point out.

AuthorsOpenReview ↗

Rebuttal response 2/N

Questions for authors

Q1. Were the evaluated Neuroprobe subjects excluded entirely from PopT-v2 and BaRISTA pretraining, or were other recording sessions from those subjects included? … provide the subject/session lists for pretraining and evaluation, identify the checkpoints used, and clarify how any overlap affects the interpretation of Table 1.

Our PopT-v2 pretraining and BaRISTA’s published pretraining include other recordings from all five evaluated Neuroprobe participants, while excluding their benchmark recordings. Both protocols therefore hold out recordings rather than participants. We will update the text to make that more apparent when making conclusions about the leaderboard results and also add Appendix N that clarifies the pretraining sessions.

Table 14 (Appendix N). Sessions used in pretraining by each model and benchmark, per participant. Entries are zero-based session IDs; “—” means no Neuroprobe session.

ParticipantPopT-v2 pretrainingBaRISTA pretrainingBenchmark target
10, 20, 21
20, 1, 2, 3, 4, 50, 1, 2, 3, 4—
31, 21, 20
41, 210
500—
60, 10, 1—
7110
800—
900—
10110

To test sensitivity to participant inclusion during pretraining, we pretrained two additional PopT variants with participant 7 or participant 10 excluded, then fine-tuned and evaluated them on that participant’s downstream Main tasks. Across three fine-tuning seeds and both folds, exclusion changed mean ROC-AUC by −0.09 ± 0.83 and +0.24 ± 0.49 percentage points, respectively (mean ± SD across seeds). These mean effects were smaller than the observed variation across fine-tuning seeds, and the exclusion effects changed direction across seeds for both participants. They were also smaller than the corresponding penalties from random initialization: 4.63 and 5.66 points. These results suggest limited aggregate sensitivity to participant exclusion in the two participants tested; individual task effects varied, and the repeated seeds assess fine-tuning rather than pretraining variability. We will clarify the pretraining participant overlap in the manuscript and explicitly qualify the Neuroprobe results as transfer to held-out recordings from previously seen participants, rather than evidence of generalization to entirely unseen participants.

Q2. Were the source recordings already filtered at 50 Hz? If not, does applying the appropriate line-noise filtering change the Pippi results?

We will update the released Pippi preprocessing configuration to use 50 Hz harmonic notches. The supplied recipe used 60 Hz notches, and our preparation pipeline contains no explicit upstream 50 Hz notch. Spectral inspection across all five benchmark participants identified residual 50 Hz peaks in three participants, so we cannot assume that line noise was adequately removed upstream.

Initial results using the corrected recipes with PopT and HTNet on three participant–task pairs, spanning three participants and three tasks, show mean performance changes smaller in magnitude than the baseline seed SD in five of six comparisons, based on three matched seeds and both folds. The exception so far is PopT on participant 42’s word-duration task (+3.39 ROC-AUC percentage points versus a baseline seed SD of 1.00). We will continue evaluating Pippi with the corrected 50 Hz preprocessing and update this response as additional results become available.

AuthorsOpenReview ↗

Rebuttal response 3/N

Q3. Could you add confidence intervals and paired model comparisons to Tables 1, 4, and 5, accounting for observations from the same subject being related?

We added score intervals and paired model comparisons for Tables 1, 4 and 5 in Appendix L. We used 50,000 bootstrap resamples of participants within each dataset, keeping their sessions, tasks and folds together and applying the same resamples to all models. On Main, PopT and DIVER show supported gains over both spectral baselines on Neuroprobe, while gains over spectral MLP are small or inconsistent on BYD and Pippi (Table R3). The intervals are pointwise and unadjusted, conditional on the evaluated runs and fixed cohort; Appendix L gives participant counts, the aggregation rule and the limits of inference.

Table R3. Main: gain over spectral MLP, ΔROC-AUC × 100 [95% CI]. A supported gain means the paired interval lies above zero.

ModelNeuroprobeBYDPippi
PopT+2.13 [1.47, 3.06]+0.63 [−0.68, 1.96]+1.21 [0.06, 2.73]
BaRISTA+1.44 [−0.02, 3.68]+0.69 [−0.92, 2.48]−0.87 [−4.25, 2.43]
DIVER+2.64 [1.86, 3.79]−2.00 [−3.76, −0.34]+0.19 [−1.68, 2.50]

All comparisons use common evaluation support, with participants resampled jointly across models (Neuroprobe: 5; BYD: 16; Pippi: 5). Appendix L reports detailed counts and exclusions.

validation class counts and show whether Main membership and model rankings are sensitive to the 0.60 threshold.

We add a report of validation class counts by dataset and subset (Table 12). With Pippi, there are 5 single-class folds, which were already excluded from the scorecards. Pippi does have a few more folds with sparse support: 12 of 85 scored Main folds contain fewer than 10 minority-class examples.

Table 12 (Appendix M). Validation class counts. Counts include all stored folds. After excluding single-class folds, Pippi has fewer than ten minority-class examples in 12/85 scored Main folds and 3/60 scored Challenge folds. The cutoff of ten is descriptive, not a validated power criterion.

DatasetSubsetVal. total / min / med / maxMedian / minorityMinority <10 / / stored foldsSingle-class / folds
NeuroprobeMain377 / 875 / 8754370 / 980
NeuroprobeChallenge485 / 875 / 8754370 / 520
BYDMain44 / 118 / 272590 / 5080
BYDChallenge44 / 114 / 176570 / 3620
PippiMain6 / 55 / 1682215 / 883
PippiChallenge6 / 76 / 133375 / 622

We evaluated the sensitivity of our Main membership to the validation threshold. Under the original 2 baseline model union rule, thresholds of 0.55, 0.60 and 0.65 select 507, 347 and 217 units, respectively, yet preserve Overall model rankings within both input tracks (Appendix M, Fig. 17). The existing task-level score plot (Fig. 5) shows where units lie relative to the original cutoff.

Reviewer_GG1nOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces iMINDBench, which aligns movie-watching iEEG datasets from three institutions to 15 audiovisual and language decoding tasks, with standardized data splits, preprocessing tracks, and a unified baseline evaluation framework.

Critical strengths

S1. Bringing previously separate multi-institution iEEG datasets under shared tasks and evaluation protocols provides a valuable benchmark for comparing model performance across heterogeneous recordings.

S2. Preprocessing ablations and simple spectral baselines show that preprocessing choices can affect performance more than model choice.

Critical weaknesses

W1. A central finding is that cross-institution supervision yields limited gains, but the evidence comes only from PopT-v2. The scope and generalizability need further clarification.

W2. Annotation quality for the downstream tasks on the newly added datasets is not quantitatively assessed.

Questions for authors

Q1. What specifically does this work contribute beyond Neuroprobe?

Q2. How reliable are the annotated labels across datasets? My main concerns are annotation quality on the added datasets (e.g., number of annotators, review procedures, and onset/offset timing errors), temporal alignment (whether 1s windows can straddle adjacent events or label boundaries), and task comparability (how Dutch labels are adapted, whether the impact of dataset differences on joint training has been evaluated).

Q3. Could the authors provide more evidence that adding training data from other institutions yields only limited gains? At the subject level, how does the current model handle subject differences and variation in electrodes, locations, and brain-region coverage? How much overlap is there in brain-region coverage between the source and target datasets? At the model level, does this finding hold for other models that support joint training across subjects? If it holds only for the current model, how should the conclusion be qualified?

4 comments
AuthorsOpenReview ↗

Rebuttal response 1/N

Thank you for your thoughtful review and for recognizing the value of a shared multi-institution benchmark and our analysis of preprocessing choices. We respond to your concerns below with additional experiments, analyses, and clarifications. We welcome further discussion and suggestions for improving the work.

W1. cross-institution supervision yields limited gains, but the evidence comes only from PopT-v2. The scope and generalizability need further clarification.

The within-dataset and multi-dataset experiments require finetuning on multiple participants which means different electrode layouts must be mapped to the same single logit output. Our other evaluated model implementations use decoding heads tied to a fixed electrode layout; extending them to joint training across participants requires adapting and validating those heads. We attempted to replace the decoder heads with methods that allow for input channel variation during finetuning (tried: cls token, pooling), but found much poorer performance. More development is needed to get a clear signal from these additional models and baselines. We instead include additional experiments with fixed budget performance given different data sources (see Q3 response).

Questions for authors

Q1. What specifically does this work contribute beyond Neuroprobe?

We contribute 2 additional annotated movie watching datasets aligned in electrode space and labels. With this multi-institution expansion, we (1) carefully benchmark several common baselines and modern pretrained models in a preprocessing aware manner and (2) conduct initial experiments on multi-subject and multi-institution finetuning’s data scaling effects on downstream dataset evaluation.

Q2. How reliable are the annotated labels across datasets? … annotation quality on the added datasets (e.g., number of annotators, review procedures, and onset/offset timing errors)

We follow the Brain Treebank annotation protocol, reusing its feature-extraction code with dataset-specific adaptations for BYD and Pippi. Quality control differs by feature type:

  • Automatically computed features (7/15 tasks)—volume, change in volume, pitch, global and local optical flow, frame brightness, and GPT-2 surprisal—use the shared extraction pipeline, with language-specific models for surprisal.

  • Transcript- and timing-derived features (5/15 tasks)—speech, sentence onset, word index, word gap, and word duration—are computed from transcripts and timings initialized using language-specific ASR models, then manually reviewed and corrected at 4× slowed playback, using both spectrograms and waveforms, following the Brain Treebank procedure.

  • Linguistic annotations (2/15 tasks)—part of speech and dependency-head position—are initialized with automatically generated parses in CoNLL-U format and corrected by an expert linguist annotator in the corresponding language.

  • Face counts (1/15 tasks) are initialized automatically and manually corrected using slowed video playback.

We are also conducting an additional co-author spot check of randomly sampled clips from both movies, focusing on timing, linguistic labels, and face counts. This check is in progress, and we will update the response with the results when available.

temporal alignment (whether 1s windows can straddle adjacent events or label boundaries)

Regarding temporal alignment, the one-second neural windows follow the Brain Treebank/Neuroprobe convention of alignment to word onsets or sampled nonverbal-interval onsets. A window can contain neighboring words or events; its label refers to the designated target event or interval. We will clarify this window-to-label relationship in the manuscript.

task comparability (how Dutch labels are adapted, whether the impact of dataset differences on joint training has been evaluated)

For Dutch, linguistic features are extracted using language-specific tools and mapped to the shared task definitions. Continuous tasks use within-session thresholds to represent relative stimulus properties, such as high versus low volume within a movie. This preserves a common task definition, while absolute values and class boundaries can differ across movies. These differences remain a source of heterogeneity in joint multi-dataset training. Our multi-dataset experiments evaluate performance under this combined language and stimulus heterogeneity; they do not isolate the contribution of each difference.

AuthorsOpenReview ↗

Rebuttal response 2/N

Q3. Could the authors provide more evidence that adding training data from other institutions yields only limited gains?

In addition to Fig. 3b, which shows that adding subjects or datasets does not consistently improve performance under the evaluated PopT protocol, we conducted a budget-aligned experiment using the same amount of training data from different sources. The comparison covers all 49 Neuroprobe Main task–session units, using two folds and seed 42, with 1,000 updates and batch size 128 in each setting. Mean performance was highest for within-session training and decreased with within-dataset and multi-dataset pooling.

Training settingMean test ROC-AUC × 100Change vs. within-session
Within-session72.15—
Within-dataset69.29−2.86 pp
Multi-dataset67.08−5.07 pp

This complements the original experiments by testing training-data source composition under a fixed data budget: pooled sampling replaces some target-session examples, whereas the original experiments retain those examples and add other sources. We will add these results to the appendix and clarify in the abstract and conclusion that the scaling findings concern PopT under the tested protocol.

how does the current model handle subject differences and variation in electrodes, locations, and brain-region coverage? How much overlap is there in brain-region coverage between the source and target datasets?

PopT-v2 represents each electrode using its signal features and LPI coordinates, and uses a shared classification-token readout to accommodate different electrode counts and layouts across participants. This supports joint training without requiring identical electrode montages, but does not eliminate differences in anatomical coverage. Appendix A, Fig. 4 illustrates the electrode distributions across datasets. We see that electrode layouts generally overlap across datasets when we have all participants electrodes pooled together.

does this finding hold for other models that support joint training across subjects? If it holds only for the current model, how should the conclusion be qualified?

Our other evaluated model implementations use decoding heads tied to a fixed electrode layout. Extending them to joint training across participants requires adapting and validating those heads. Consequently, the present scaling results concern PopT under the tested protocol; they do not establish the scaling behavior of other architectures.

Thanks for the additional experiments and for clarifying that the scaling findings apply to PopT under the tested protocol. The response addresses some concerns, but validation of label reliability and analysis supporting the cross-institution scaling findings remain limited, so I maintain my original score.

AuthorsOpenReview ↗

Reviewer response

Dear Reviewer,

Thank you for clarifying your remaining concerns and for your patience while we completed the label-quality spot checks. We now report results from two reviewers independent of the original annotator and provide additional supplementary materials that make the annotations directly inspectable. Below, we also further clarify the scope of our multi-dataset scaling findings.

Label reliability. Two reviewers independent of the original annotator reviewed different sample sets, each containing ten random 5-second windows per dataset (20 distinct windows per dataset in total, with each window having some subset of transcription, timing, linguistic, face count features to review).

Annotation checksConfirmed / reviewedConfirmation rate
Transcription, timing, sentence start, and nonverbal intervals79/79100%
Face counts33/3594.3%
All checks112/11498.2%

The independent reviewers confirmed 112 of 114 completed checks (98.2%), including all 79 completed speech-related checks. Two Pippi face counts were marked for correction (3 → 2 faces; 2 → 0 faces), where characters turned around or became occluded temporarily during the clip. Timing confirmation required both boundaries to be within 0.25 seconds.

Appendix R includes the detailed results (Table 16), verification procedure, and a screenshot of the annotation-review interface (Figure 20). For further third-party inspection, the updated supplementary materials provide the complete BYD and Pippi transcripts, timings, and features as downloadable CSVs and an offline, searchable webpage, alongside example screenshots showing aligned video frames, transcripts, waveforms, and spectrograms. To explore these materials, download and extract the updated supplementary ZIP, then double-click movie_feature_review/index.html. The viewer works locally without installation.

Cross-institution scaling. We clarify that our findings concern PopT-v2 under the tested protocol. The reason for this is that other evaluated model implementations do not natively support joint fine-tuning across heterogeneous electrode layouts. Our attempts to adapt their decoding heads using CLS-token decoding or attention pooling reduced performance, motivating our use of PopT-v2 for this analysis.

Within this scope, our experiments show that adding subjects or datasets does not consistently improve performance. These results motivate developing architectures and training methods that more effectively accommodate heterogeneous electrode layouts and benefit from additional heterogeneous data.

We appreciate your time and consideration and hope these additions address your remaining concerns about annotation reliability and the interpretation of the scaling results.

Reviewer_ZKGwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents IMINDBENCH, a multi-institution iEEG neural decoding benchmarks. It harmonize targets and electrode metadata and aligns the BYD and Pippi datasets to follow the same preprocessing and task setup of Neuroprobe dataset. The paper does within-session, within-dataset, and multi-dataset scaling experiment, and benchmarking on models with two main input tracks (Multi-STFT track and Waveform track). The paper finds that adding data from other institutions provide little benefits and that engineered spectral baseline remains competitive across institutions compared to pre-trained foundation model.

Critical strengths
  1. The data harmonization and task alignments are valuable: Harmonizing the BYD and Pippi dataset into the same format as Neuroprobe and aligning the tasks are non trivial works. The resulting IMINDBENCH is a valuable multi-institution neuroal decoding benchmark for testing model generalizability.

  2. I personally really like that the paper pays the attention to the different preprocessing settings. The paper tests different normalization and high-passs filtering setting, which is commonly being ignored by the field. Preprocessing can affect the downstream performance alot and I like the analysis.

Critical weaknesses
  1. Cross-institution evaluation is not well-defined: One of the key motivation of the paper is to evaluate model capability across institutions, however the main experiment is not experimenting the true setting of "leave-one-instituion-off" setting, which best align with the problem formulation: we want to know how model generalize across unseen data sources.

  2. Task harmonization does not fully resolve label heterogeneity: When we want to evaluate cross -dataset generalizability, there are actually two major source of domain shifts: the institution / stimulus signal shift, and the label shifts. For iEEG foundation models, we are commonly refer to the first: we want the model to be generalizable / transferrable across different data sources, while still evaluating using same kind of labels. In this paper, I have concerns regarding the labels, as they might be defining differently across datasets. For example, the paper states that "Neuroprobe uses bottom-versus-top quartiles, while BYD and Pippi use bottom-versus-top terciles". Also, for tasks like volume and brightness, in different dataset/film, same "high" volume label does not necessarily mean the absolute volume is alined. In addition, Neuroprobe and BYD's source is in English, while Pippi is in Dutch: how while, for example, for GPT-2 to be able to handle it? All of the above makes it difficult to analyze whether the model is learning to generalize across different data sources, or to generalize across different label definitions.

  3. Experiment setting needs refinement: The major data scaling experiment for within-dataset and multi-dataset only uses PopT. We should make conclusion when using more baselines. Also it is not clearly explained how we sample different data, how the batch is composited when performing multi-dataset training. It will also be helpful to conduct budget-aligned experiment (using same amount of data but from different sources).

In addition, the setting for different baseline model is not unified: For example, Diver-1 is not using normalization, is completely frozen, and using the "tiny" variants, whereas PopT is being fine-tuned and has normalization.

  1. Regarding the definition of the "Main subset": Testing the actual "decodability" is a good move, but the paper is using two model's performance on the data to determine the decodability and to select the main subset and challenging subset. This selection method has model-dependent selection bias and might favor certain type of data when constructing the main dataset. More illustration / explanation of the method is needed.

  2. Statistically importance: In table one, many metrics are only differed by 0.02. And in figure 9, some tasks as unique supported subjects as few as 1. In figure 12, we see that tuning hyperparamter of PopT can also leads to performance improvement. All the above raise concerns that how reliable the benchmark conclusion is.

  3. Wordings: The paper uses "general-purpose iEEG foundation models" a lot across sections. However, I want to point out that all the tasks in the benchmark is inherit from Neuroprobe and is not truly "general-purpose": there is no memory decoding, motor, clinical, etc. I would suggest either to clearly define "general-purpose" or narrow the claims.

Questions for authors

I believe I have put all my concerns and questions in the weakness section. I would appreciate author to address them.

3 comments
AuthorsOpenReview ↗

Rebuttal response 1/N

Thank you for your thoughtful review and for recognizing our contributions to multi-dataset harmonization and the importance of preprocessing choices when comparing models. We respond to your concerns below with additional experiments, analyses, and clarifications. We welcome further discussion and suggestions for improving the work.

Q1: Cross-institution evaluation is not well-defined … not experimenting the true setting of "leave-one-instituion-off" setting

Our main benchmark evaluates cross-institution transfer from pretraining to supervised finetuning: models pretrained on external recordings are adapted using labeled training data from each target participant and evaluated on held-out examples. BYD and Pippi introduce two institutions absent from the evaluated models’ pretraining data, allowing us to assess how well their pretrained representations transfer to these new datasets.

Q2: Task harmonization does not fully resolve label heterogeneity

Our harmonization aligns task definitions across datasets while retaining stimulus-relative labels. The main benchmark evaluates pretrained models after finetuning on each target participant’s data, so training and evaluation use the same target label definition.

  • Quartiles versus terciles: BYD and Pippi use terciles to retain enough labeled examples per split from their shorter recordings. This trades a consistent quantile cutoff across datasets for greater sample support: including less extreme examples can make the low-versus-high distinction harder. The labels retain their stimulus-relative interpretation, but task difficulty may differ across datasets.

  • Absolute stimulus values: “High” volume or brightness is defined relative to each movie. Physical sound levels and screen luminance also depend on playback settings and recording-room conditions, for which we lack calibrated measurements. We therefore use stimulus-relative labels and do not assume matched absolute exposure across institutions.

  • English versus Dutch: Pippi surprisal is computed using Dutch GPT-2 (GroNLP/gpt2-small-dutch), while BYD uses English GPT-2. High and low surprisal labels are defined by within-movie quantiles, preserving the interpretation of relatively unexpected versus predictable words without requiring raw surprisal values to be calibrated across languages.

In the multi-dataset experiments, models must additionally accommodate these differences in label distributions. Those results therefore reflect transfer across the combined differences in participants, recordings, stimuli, languages, and labels. We will clarify that they do not isolate the effect of institution alone.

AuthorsOpenReview ↗

Rebuttal response 2/N

Q3: The major data scaling experiment for within-dataset and multi-dataset only uses PopT. We should make conclusion when using more baselines.

We attempted to evaluate multi-dataset results with additional models, but found that significant changes to the proposed models were necessary to allow for competitive performance. The within-dataset and multi-dataset experiments require finetuning on multiple participants which means different electrode layouts must be mapped to the same single logit output. Our other evaluated model implementations use decoding heads tied to a fixed electrode layout; extending them to joint training across participants requires adapting and validating those heads. We attempted to replace the decoder heads with methods that allow for input channel variation during finetuning (tried: cls token, pooling), but found much poorer performance. More development is needed to get a clear signal from these additional models and baselines.

how we sample different data: how the batch is composited when performing multi-dataset training

For each task, pooled training uses the eligible task-decodable sessions within the target dataset or across datasets. We cap each source session at the target session’s training-window count, then shuffle windows from the combined pool into batches; batches are not balanced equally across datasets. Validation and test remain on the target session. We will clarify this in the manuscript.

conduct budget-aligned experiment

We conducted the requested budget-aligned experiment, comparing PopT trained on the same amount of data drawn from within-session, within-dataset, or multi-dataset sources. The comparison covers all 49 Neuroprobe Main task–session units, using two folds and seed 42. For each target and fold, all three settings use the same number of training windows, followed by 1,000 updates with batch size 128.

Training settingMean test ROC-AUC × 100Change vs. within-session
Within-session72.15—
Within-dataset69.29−2.86 pp
Multi-dataset67.08−5.07 pp

Pooling did not improve average performance under matched data and update budgets. Because pooled sampling replaces some target-session examples with examples from other sessions, this comparison tests source composition under a fixed data budget. It complements the paper’s original experiments, which retain all target-session training examples and add data from other subjects or datasets. We will add this comparison to the appendix to characterize how training-data source affects performance when the amount of training data is held fixed.

Q4: Diver-1 is not using normalization, is completely frozen, and using the "tiny" variants, whereas PopT is being fine-tuned and has normalization.

We selected DIVER-1 Tiny because its 0.1-second patches provide ten temporal patches per channel within our one-second inputs; the larger public Small checkpoint uses one-second patches. We froze its backbone following DIVER-1’s published Neuroprobe results, where freezing achieved higher average ROC-AUC than full finetuning (67.6 versus 66.2) DIVER-1. Similarly, we retain PopT’s proposed finetuning recipe.

We allow model-specific normalization to maintain consistency with pretraining and avoid introducing an input-distribution mismatch. Supporting this choice, HTNet run on normalized vs scaled versions of the data showed small differences: −0.01, +0.70, and −0.05 ROC-AUC percentage points on matched Neuroprobe, BYD, and Pippi cases, respectively. We will clarify that our comparisons evaluate models together with their preprocessing and adaptation recipes.

Q5: This selection method has model-dependent selection bias and might favor certain type of data when constructing the main dataset. More illustration / explanation of the method is needed.

Main is a model-dependent screening subset: a participant–session–task unit is included when either the Multi-STFT or HTNet selector exceeds mean validation ROC-AUC 0.60. Of the 347 selected units, 215 pass both selectors, 87 pass only Multi-STFT, and 45 pass only HTNet. The union therefore retains cases missed by either selector alone. Membership is determined using validation performance, without using test scores.

To characterize the Main subset threshold boundary, we report threshold sensitivity in Appendix M: union thresholds of 0.55, 0.60 and 0.65 select 507, 347 and 217 units, respectively. Despite this change in threshold, Overall rankings are preserved within both input tracks (Fig. 17), which shows robustness of this thresholding approach. The task-level screening-score plot (Fig. 5), validation class counts (Table 12), and All/Challenge scorecards make the selection rule and its limits explicit.

AuthorsOpenReview ↗

Rebuttal response 3/N

Q6: In table one, many metrics are only differed by 0.02. … All the above raise concerns that how reliable the benchmark conclusion is.

We added paired participant-bootstrap 95% confidence intervals to distinguish supported gains from small score differences. On Main, 2 of 3 pretrained models (PopT and DIVER) show supported gains over both spectral MLP and CNN on Neuroprobe. Against spectral MLP, 0 of 3 show supported gains on BYD and 1 of 3 on Pippi, supporting the competitiveness of a strong spectral baseline on these datasets. Table R1 gives the paired effects, and Appendix L in the updated paper now reports the complete Main, All and Challenge comparisons.

Table R1. Main: gain over spectral MLP, ΔROC-AUC × 100 [95% CI]. A supported gain means the paired interval lies above zero.

ModelNeuroprobeBYDPippi
PopT+2.13 [1.47, 3.06]+0.63 [−0.68, 1.96]+1.21 [0.06, 2.73]
BaRISTA+1.44 [−0.02, 3.68]+0.69 [−0.92, 2.48]−0.87 [−4.25, 2.43]
DIVER+2.64 [1.86, 3.79]−2.00 [−3.76, −0.34]+0.19 [−1.68, 2.50]

In figure 9, some tasks as unique supported subjects as few as 1.

Thank you for highlighting this issue. Six of 44 dataset–task combinations in Fig. 9 have only one supported participant. Excluding these combinations preserves the overall interpretation of modest, dataset-dependent gains from pooled supervision under the evaluated PopT protocol.

Within-dataset pooling continues to improve BYD, and the incremental benefit of multi-dataset over within-dataset training remains uncertain in all three datasets, with all intervals spanning zero before and after exclusion. The main changes are a small positive multi-dataset-versus-within-session effect on Neuroprobe and a positive but still uncertain mean on Pippi.

Training comparisonDatasetOriginal cohortAfter exclusion
Within-dataset − within-sessionNeuroprobe+0.12 [−0.42, +0.75]+0.34 [−0.01, +0.80]
BYD+1.42 [+0.80, +2.06]+1.42 [+0.80, +2.06]
Pippi+0.43 [−0.01, +0.92]+0.42 [−0.19, +1.10]
Multi-dataset − within-sessionNeuroprobe+0.11 [−0.58, +0.54]+0.36 [+0.25, +0.58]
BYD+1.53 [+0.91, +2.06]+1.53 [+0.91, +2.06]
Pippi−0.30 [−2.05, +2.16]+0.85 [−1.83, +3.75]

Mean changes in ROC-AUC percentage points [95% participant-bootstrap CI].

We will update the scaling figures to use participant-bootstrap intervals and retain single-participant task estimates without participant-level intervals, explicitly noting their limited support.

In figure 12, we see that tuning hyperparamter of PopT can also leads to performance improvement

Figure 12 shows that adaptation hyperparameters can materially affect performance. Our main comparisons use fixed, documented model-specific recipes, so their rankings describe performance under those settings rather than each model’s best achievable performance. We will make this qualification explicit and present the PopT tuning experiment as evidence of optimization sensitivity.

Q7: all the tasks in the benchmark is inherit from Neuroprobe and is not truly "general-purpose" … clearly define "general-purpose" or narrow the claims.

“General-purpose” reflects our aim to develop a benchmark with progressively broader task coverage. The current release expands evaluation across institutions and participants within audiovisual and linguistic decoding during movie watching. We will distinguish this current scope from the longer-term goal of incorporating additional domains, such as memory, motor, and clinical tasks.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Meta response

We thank the reviewers for their careful reading and constructive feedback, as well as continued engagement. The reviews recognize the value of harmonizing multi-institution iEEG datasets and accounting for preprocessing when comparing models. We appreciate the reviewers comments which have helped us better characterize this benchmark through additional experiments, uncertainty analyses, and clearer statements of the benchmark’s scope. The main updates are summarized below.

  • Scope and interpretation. We clarified that the "cross-institution" nature of this benchmark focuses on transfer from pretraining to supervised finetuning on movie-watching tasks, with stimulus-relative labels. This is important, since it is the most common setting in which brain foundation models are used: we want to benchmark the use pattern of a practitioner who takes off-the-shelf weights and fine-tunes them for their experiment. We qualified the scaling conclusions as specific to PopT under the evaluated protocol and explained model-specific preprocessing and adaptation choices. (Abstract, Introduction, Conclusion; R1 Q1–Q4 and Q7; R2 W1/Q3.)

  • Statistical reliability. We added participant-bootstrap 95% confidence intervals and paired model comparisons for Main, All, and Challenge, accounting for related observations from the same participant. These support clearer pretrained-model gains on Neuroprobe and the competitiveness of spectral baselines on BYD and Pippi. We also updated the scaling intervals and checked sensitivity to excluding single-participant tasks. (Appendix L; Figs. 3 and 9; R1 Q6 and R3 Q3.)

  • Main-subset selection. We documented the contributions of both screening models. Thresholds of 0.55, 0.60, and 0.65 retain 507, 347, and 217 units while preserving Overall rankings within each input track; membership and some dataset-specific rankings change. (Appendix M; R1 Q5 and R3 Q3.)

  • Fixed training budgets across data types. We added a fixed-data-budget comparison covering all 49 Neuroprobe Main units: within-dataset and multi-dataset pooling reduce mean ROC-AUC by 2.86 and 5.07 percentage points relative to within-session training. We also documented target-session exposure and checked training through 4,000 updates on a selected panel, finding limited benefit under the tested recipe. (Appendix O; R1 Q3, R2 Q3, and R3 Q4.)

  • Pretraining participant overlap. We documented pretraining and evaluation recordings and clarified that Neuroprobe holds out recordings rather than participants. For completion, we add two participant-excluded PopT variants, evaluated across three finetuning seeds, show mean changes of −0.09 and +0.24 percentage points, alongside random-initialization comparisons (Appendix N; R3 Q1.)

  • Annotation quality and label comparability. We detailed our use of the Brain Treebank extraction pipeline, language-specific tools, and manual review of timing, linguistic annotations, and face counts. We explained the quantile-label tradeoff and differences in absolute stimulus values. (R1 Q2; R2 Q2.) Two independent, human reviewers of the original annotator confirmed 112/114 completed random spot checks (98.2%) of labels across 20 distinct five-second windows per dataset, supporting high agreeability about label quality. Supplementary CSVs and an offline viewer make the BYD/Pippi annotations further inspectable (Appendix R; R2 follow-up).

  • Pippi line-noise filtering. We completed 50 Hz versus 60 Hz comparisons across all 15 tasks, five participants, and eight model/input combinations. Aggregate changes are below one ROC-AUC percentage point across Main, All, and Challenge; all Main paired confidence intervals include zero (Appendix P; R3 follow-up.)

  • Temporal boundaries. We clarified that the reported 15-second preprocessing context can cross split boundaries and propose to move towards 1s preprocessing windows with temporal gaps enforced between splits. We completed Main-subset reruns of six model/input combinations across Neuroprobe, BYD and Pippi (both folds, base seed 42), using one-second preprocessing with disjoint raw support and preserving all original test examples. Dataset-level rankings are unchanged among these six models, and spectral MLP remains competitive on BYD and Pippi. Averaged equally over four waveform models and three datasets, the change from the reported pipeline is −0.75 AUROC percentage points (paired participant-bootstrap 95% CI [−1.07, −0.47]; Appendix Q.2, Table 16). The corresponding scorecards and figures will be updated with the evaluations under this new preprocessing and temporal gap rule.

We thank the reviewers and the AC for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.