- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces iMINDBench, which extends Neuroprobe’s 15 naturalistic iEEG decoding tasks to two additional movie-watching datasets: Bang! You’re Dead from Cedars-Sinai and Pippi from Utrecht. It aligns stimulus labels and electrode metadata, provides fixed evaluation splits, and defines spectral and waveform preprocessing tracks, including documented model-specific variations. The experiments compare conventional decoders with pretrained models and examine whether adding labeled data from other subjects and institutions improves performance. The main findings are that preprocessing choices have a substantial effect, a simple Multi-STFT MLP also remains competitive with the pretrained waveform systems, and PopT-v2 gains relatively little from broader supervision.
Critical strengths
-
The paper makes a convincing case for taking preprocessing seriously when comparing models, and this is something that previous benchmarks, in my opinion, might have skipped. In Section 4.2 the authors show that normalization alone can change baseline ROC-AUC by more than 0.10 in some settings. Removing the 0.5 Hz high-pass filter also reduces HTNet’s average performance from 0.612 to 0.589, with decreases across all three datasets. These changes can exceed the differences between models in Table 1. I appreciate that the paper distinguishes comparisons using the same inputs from comparisons between complete processing pipelines, and asks for compatible baseline reruns when preprocessing changes. This makes the benchmark more useful for understanding where improvements come from.
-
Bringing these datasets into a common evaluation framework is a useful contribution. The datasets differ in their stimuli, languages, electrode coverage, and recording setups, so aligning their labels and electrode information requires substantial work. The shared task definitions, fixed splits, and reporting of preprocessing and pretraining data provide a practical basis for comparing models across institutions. If the code and supporting materials are released as described, this could make evaluation on multiple iEEG datasets considerably easier for other researchers.
Critical weaknesses
-
Now, some of the conclusions that are made in the paper are quite a bit stronger than the evidence that currently supports them. Tables 1, 4, and 5 do not include confidence intervals or paired comparisons, making it difficult to judge whether small differences between models are reliable. The stability of the Main/Challenge grouping is also unclear, particularly for the shorter recordings, where relatively few validation examples may determine whether a unit passes the 0.60 threshold. The scaling analysis includes confidence intervals, but its cross-institution results are based solely on PopT-v2. This limitation is acknowledged in Section 7 and should also be reflected in the abstract and conclusion. The training setup needs clarification as well: Appendix I specifies 1,000 fine-tuning updates, but it is unclear whether this budget is used in every scaling setting and how often the target session is sampled as more data are added.
-
Several protocol details still need clarification, despite the useful code release. It remains unclear whether the evaluated Neuroprobe subjects contributed other sessions to pretraining; this matters for interpreting generalization, although it would not automatically invalidate the stated evaluation. The supplied Pippi pipelines use 60 Hz notches, leaving open whether 50 Hz line noise was addressed upstream. The BYD/Pippi split builders also lack an explicit temporal gap, while some preprocessing uses surrounding signal from the full recording. Finally, some supplied defaults differ from the manuscript or earlier settings described in the README.
Questions for authors
-
Were the evaluated Neuroprobe subjects excluded entirely from PopT-v2 and BaRISTA pretraining, or were other recording sessions from those subjects included? Please provide the subject/session lists for pretraining and evaluation, identify the checkpoints used, and clarify how any overlap affects the interpretation of Table 1.
-
The supplied Pippi pipelines use 60 Hz notches. Were the source recordings already filtered at 50 Hz? If not, does applying the appropriate line-noise filtering change the Pippi results?
-
Could you add confidence intervals and paired model comparisons to Tables 1, 4, and 5, accounting for observations from the same subject being related? Please also report validation class counts and show whether Main membership and model rankings are sensitive to the 0.60 threshold.
-
Is the same 1,000-update fine-tuning budget used for within-session, within-dataset, and multi-dataset training? How are examples sampled in each setting, and how much exposure does the model receive to the target session as the pooled dataset grows?
-
The supplied BYD/Pippi split builders do not appear to include a temporal exclusion gap, while the waveform preprocessing can load context across split boundaries. Was an additional safeguard used for the reported experiments, and are the results stable when a suitable gap is introduced?
10 comments
Rebuttal response 2/N
Questions for authors
Q1. Were the evaluated Neuroprobe subjects excluded entirely from PopT-v2 and BaRISTA pretraining, or were other recording sessions from those subjects included? … provide the subject/session lists for pretraining and evaluation, identify the checkpoints used, and clarify how any overlap affects the interpretation of Table 1.
Our PopT-v2 pretraining and BaRISTA’s published pretraining include other recordings from all five evaluated Neuroprobe participants, while excluding their benchmark recordings. Both protocols therefore hold out recordings rather than participants. We will update the text to make that more apparent when making conclusions about the leaderboard results and also add Appendix N that clarifies the pretraining sessions.
Table 14 (Appendix N). Sessions used in pretraining by each model and benchmark, per participant. Entries are zero-based session IDs; “—” means no Neuroprobe session.
| Participant | PopT-v2 pretraining | BaRISTA pretraining | Benchmark target |
|---|---|---|---|
| 1 | 0, 2 | 0, 2 | 1 |
| 2 | 0, 1, 2, 3, 4, 5 | 0, 1, 2, 3, 4 | — |
| 3 | 1, 2 | 1, 2 | 0 |
| 4 | 1, 2 | 1 | 0 |
| 5 | 0 | 0 | — |
| 6 | 0, 1 | 0, 1 | — |
| 7 | 1 | 1 | 0 |
| 8 | 0 | 0 | — |
| 9 | 0 | 0 | — |
| 10 | 1 | 1 | 0 |
To test sensitivity to participant inclusion during pretraining, we pretrained two additional PopT variants with participant 7 or participant 10 excluded, then fine-tuned and evaluated them on that participant’s downstream Main tasks. Across three fine-tuning seeds and both folds, exclusion changed mean ROC-AUC by −0.09 ± 0.83 and +0.24 ± 0.49 percentage points, respectively (mean ± SD across seeds). These mean effects were smaller than the observed variation across fine-tuning seeds, and the exclusion effects changed direction across seeds for both participants. They were also smaller than the corresponding penalties from random initialization: 4.63 and 5.66 points. These results suggest limited aggregate sensitivity to participant exclusion in the two participants tested; individual task effects varied, and the repeated seeds assess fine-tuning rather than pretraining variability. We will clarify the pretraining participant overlap in the manuscript and explicitly qualify the Neuroprobe results as transfer to held-out recordings from previously seen participants, rather than evidence of generalization to entirely unseen participants.
Q2. Were the source recordings already filtered at 50 Hz? If not, does applying the appropriate line-noise filtering change the Pippi results?
We will update the released Pippi preprocessing configuration to use 50 Hz harmonic notches. The supplied recipe used 60 Hz notches, and our preparation pipeline contains no explicit upstream 50 Hz notch. Spectral inspection across all five benchmark participants identified residual 50 Hz peaks in three participants, so we cannot assume that line noise was adequately removed upstream.
Initial results using the corrected recipes with PopT and HTNet on three participant–task pairs, spanning three participants and three tasks, show mean performance changes smaller in magnitude than the baseline seed SD in five of six comparisons, based on three matched seeds and both folds. The exception so far is PopT on participant 42’s word-duration task (+3.39 ROC-AUC percentage points versus a baseline seed SD of 1.00). We will continue evaluating Pippi with the corrected 50 Hz preprocessing and update this response as additional results become available.
Rebuttal response 3/N
Q3. Could you add confidence intervals and paired model comparisons to Tables 1, 4, and 5, accounting for observations from the same subject being related?
We added score intervals and paired model comparisons for Tables 1, 4 and 5 in Appendix L. We used 50,000 bootstrap resamples of participants within each dataset, keeping their sessions, tasks and folds together and applying the same resamples to all models. On Main, PopT and DIVER show supported gains over both spectral baselines on Neuroprobe, while gains over spectral MLP are small or inconsistent on BYD and Pippi (Table R3). The intervals are pointwise and unadjusted, conditional on the evaluated runs and fixed cohort; Appendix L gives participant counts, the aggregation rule and the limits of inference.
Table R3. Main: gain over spectral MLP, ΔROC-AUC × 100 [95% CI]. A supported gain means the paired interval lies above zero.
| Model | Neuroprobe | BYD | Pippi |
|---|---|---|---|
| PopT | +2.13 [1.47, 3.06] | +0.63 [−0.68, 1.96] | +1.21 [0.06, 2.73] |
| BaRISTA | +1.44 [−0.02, 3.68] | +0.69 [−0.92, 2.48] | −0.87 [−4.25, 2.43] |
| DIVER | +2.64 [1.86, 3.79] | −2.00 [−3.76, −0.34] | +0.19 [−1.68, 2.50] |
All comparisons use common evaluation support, with participants resampled jointly across models (Neuroprobe: 5; BYD: 16; Pippi: 5). Appendix L reports detailed counts and exclusions.
validation class counts and show whether Main membership and model rankings are sensitive to the 0.60 threshold.
We add a report of validation class counts by dataset and subset (Table 12). With Pippi, there are 5 single-class folds, which were already excluded from the scorecards. Pippi does have a few more folds with sparse support: 12 of 85 scored Main folds contain fewer than 10 minority-class examples.
Table 12 (Appendix M). Validation class counts. Counts include all stored folds. After excluding single-class folds, Pippi has fewer than ten minority-class examples in 12/85 scored Main folds and 3/60 scored Challenge folds. The cutoff of ten is descriptive, not a validated power criterion.
| Dataset | Subset | Val. total / min / med / max | Median / minority | Minority <10 / / stored folds | Single-class / folds |
|---|---|---|---|---|---|
| Neuroprobe | Main | 377 / 875 / 875 | 437 | 0 / 98 | 0 |
| Neuroprobe | Challenge | 485 / 875 / 875 | 437 | 0 / 52 | 0 |
| BYD | Main | 44 / 118 / 272 | 59 | 0 / 508 | 0 |
| BYD | Challenge | 44 / 114 / 176 | 57 | 0 / 362 | 0 |
| Pippi | Main | 6 / 55 / 168 | 22 | 15 / 88 | 3 |
| Pippi | Challenge | 6 / 76 / 133 | 37 | 5 / 62 | 2 |
We evaluated the sensitivity of our Main membership to the validation threshold. Under the original 2 baseline model union rule, thresholds of 0.55, 0.60 and 0.65 select 507, 347 and 217 units, respectively, yet preserve Overall model rankings within both input tracks (Appendix M, Fig. 17). The existing task-level score plot (Fig. 5) shows where units lie relative to the original cutoff.
Rebuttal response 1/N
Thank you for your thoughtful review and for recognizing the importance of preprocessing-aware comparisons and a common evaluation framework across datasets. We respond to your concerns below with additional experiments, uncertainty analyses, and protocol clarifications. We welcome further discussion and suggestions for improving the work.
We added participant-bootstrap confidence intervals and paired model comparisons; see our response to Q3 and Appendix L. We find that the confidence intervals support the existing patterns seen in the benchmarks leaderboards.
We agree that Main membership depends on the screening rule and validation support. Our Q3 response and Appendix M report a threshold sweep: Overall rankings within each input track are stable across the tested union thresholds.
We attempted to evaluate multi-dataset results with additional models, but found that significant changes to the proposed models were necessary to allow for competitive performance. The within-dataset and multi-dataset experiments require finetuning on multiple participants which means different electrode layouts must be mapped to the same single logit output. Our other evaluated model implementations use decoding heads tied to a fixed electrode layout; extending them to joint training across participants requires adapting and validating those heads. We attempted to replace the decoder heads with methods that allow for input channel variation during finetuning (tried: cls token, pooling), but found much poorer performance. More development is needed to get a clear signal from these additional models and baselines. We will clarify this in the abstract and conclusion.
We address these in our response to Q4.
Other sessions from evaluated Neuroprobe participants were included in PopT-v2 pretraining. We added participant-exclusion results and recording/checkpoint provenance in Appendix N (Table 13); see the detailed response to Q1 below.
We will correct the released Pippi configuration to use 50 Hz harmonic notches; see Q2 below for the spectral checks and initial three-seed performance comparisons.
We follow Neuroprobe’s contiguous-block splitting convention, without an additional temporal exclusion gap. Preprocessing context can cross split boundaries, but are cropped to one-second target windows. We clarify this distinction in our response to Q5 below and will document it in the manuscript.
We reviewed the supplied code against the manuscript and saved experiment configurations. We confirmed that the supplied Logistic and MLP tolerance defaults differ from those used for the reported Multi-STFT experiments. We will provide explicit reproduction configurations matching the saved settings and clarify their use in the README. We welcome any additional code–manuscript discrepancies the reviewer could point out.