- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Despite rapid advancements in machine learning (natural language, vision), decoding neural activity remains a challenging task. Indeed, small-scale, task-tailored solutions (e.g., simple CNNs) are still hard to beat by large-scale foundational models, despite their large parameter counts and associated compute demand in pretraining. This paper aims to advance such foundational models for decoding intracranial electroencephalogram (iEEG), a modality that promises high signal-to-noise ratio that may open the opportunity for high accuracy decoding. The proposed solution, DIVER-1, features invariance to the number and ordering of the channels as well as the time window, enabled through a novel architectural components and spatio-temporal resampling during self-supervised pretraining. DIVER-1 makes two contributions:
- Architecture: For computing the input embeddings, the method integrates multi-view input features (CNN features, multi-spectral embeddings, optionally position embeddings) as well as a novel Spatio-Temporal Conditional Positional Embedding (STCPE), which is a content-dependent positional bias that promises to be channel-permutation-equivariant. These inputs are then fed into an any-variate Transformer block (MOIRAI; Woo et al., 2024), which computes full attention over all electrode-time tokens. One of the advantages of DIVER-1 is its flexibility in handling different input time span and numbers of channels as well as the supporting cases where the channel information (location) is not available. This is partially reflected in pretraining, where a spatio-temporal resampling (STR) samples the spatio-temporal input dimensions during training.
- Large-scale pretraining: DIVER-1 is pretrained on ECoG and sEEG datasets with 352k channel-hours from 37 subjects. The dataset consists of a publicly available (AJILE12; Peterson et al., 2022) and a private (25 drug-resistant epilepsy patients) recording . Besides pretraining the base models and evaluating their downstream performance on iMINDBench and seizure detection (MAYO), the paper conducts a scaling study (Kaplan et al., 2020).
Overall, the paper makes the following findings:
- DIVER-1 slightly outperforms all traditional and foundational baselines on iMINDBench (0.648 vs. 0.645 against strongest PopT-v2) and MAYO (0.974 vs 0.964 against leaderboard reimpl. Olson & Mingle; no channel info available to my knowledge).
- Scaling experiments find that the training loss can generally be reduced with additional data and compute, but this does not necessarily transfer to improved downstream performance.
Critical strengths
I find the insights from the scaling experiments insightful and informative, particularly the non-faithful transfer between pretraining loss and downstream benchmark performance. Besides asking for more high-quality data, the conclusion mainly asks for new design choices that allow a transfer from pretraining to downstream tasks. While language may not be fully immune against such faithful transfers either, their next token prediction pretraining objective is highly effective and simple. Multivariate time-series do not have such a clear (established) notion of pretraining to downstream transfer.
Critical weaknesses
I'm missing a fair comparison between the proposed DIVER-1 and prior foundational models. Having introduced a new architecture (or at least new architectural components) and a new pretraining pipeline featuring significantly more data (closed-source), it is difficult to attribute the observed gains accordingly. An equal-size PopT-v2 architecture may perform as well as, or ever better than, DIVER-1 when trained with the same data. I guess the most straight-forward comparison would be to train DIVER-1 on Brain TreeBank with a similar procedure as PopT-v2.
Questions for authors
- How does DIVER-1 compare to PopT-v2 when equalizing the pretraining (dataset and compute) and the number of parameters?
- DIVER-1 allows to exclude the channel location information at the input. I am not sure where this is demonstrated. Is it MAYO? How much would the performance drop if the channel information were removed in iMINDBench? How does the performance compare to natively channel-free foundational models, such as MVPFormer?
- Having full spatial-time attention is definitely desirable; however, it also comes with additional costs. Can you quantify the cost compared to other equally-sized EEG foundation models?
- For the scaling experiments: How does the test loss behave with respect to the number of subjects? I only can see downstream performance in Figure 3b.
- What is the input to the STCPE computation (Line 160)? It cannot be X as this is itself a function of E_STCPE.
- Line 230 talks about sampling less than 30 temporal patches. I guess this changes in the 0.1s patch model.
9 comments
Response to Reviewer Ect3 [2/3]
[Q2] Channel-free operation
"I am not sure where this is demonstrated. Is it MAYO? How much would the performance drop if the channel information were removed in iMINDBench? How does the performance compare to natively channel-free foundational models, such as MVPFormer?"
Yes — MAYO is the demonstration: the dataset provides no electrode locations, so DIVER-1 runs fully channel-free there. Our new clinical evaluation (revision Appendix D.2, uploaded today) is a second one: no model receives electrode coordinates, and DIVER-1 is the best frozen encoder on both tasks.
On the performance drop: Table 1 contains the strongest version of this test — models pretrained entirely without the channel 3D-position or sub-modality embeddings lose only 0.6–0.8 AUROC points on iMINDBench.
On MVPFormer: the iMINDBench authors examined MVPFormer and did not include it in the benchmark, because its released configuration encodes each channel in five-second segments with 25 segments of temporal context, which does not map onto the benchmark's one-second decoding windows without a validated adaptation — retaining the released context provides information beyond the benchmark window, while rescaling the input changes the signal presented to the pretrained encoder (Chau et al., 2026, Appendix I.1). The same constraint applies to any comparison we could run: a fair MVPFormer evaluation on iMINDBench is not currently defined. We therefore demonstrate channel-free operation directly, as above.
[Q3] Cost of full spatio-temporal attention
"Can you quantify the cost compared to other equally-sized EEG foundation models?"
Absolute training cost is already reported in Table 2 (total FLOPs per epoch for every model size; e.g., DIVER-1-1s Small, 51M params: 256.4 PFLOPs/epoch — consistent with the DeepSpeed profiler-measured 67.2 GMACs forward per sample × 636,480 samples × 3 for the backward pass). What Table 2 does not break out is the cost of full spatio-temporal attention specifically. At the STR-capped grid (32 electrodes × 30 patches = 960 tokens), the quadratic attention interactions — the only part a factorized design removes — contribute ≈ 17% of forward compute (11.3 of 67.2 GMACs for Small); the Q/K/V/O projections, which a factorized design retains, add a further ≈ 18%. A factorized spatial+temporal design on the same grid would need ≈ 0.7 GMACs for its interactions, so full attention makes DIVER-1 ≈ 1.2× the cost of an equally-sized factorized model. This overhead is bounded by the STR token cap and grows only with longer windows or more electrodes.
[Q4] Test loss vs number of subjects
"How does the test loss behave with respect to the number of subjects? I only can see downstream performance in Figure 3b."
Agreed — this is worth reporting. Test losses across subject subsets were not directly comparable in our original setup because each run was validated on a split of its own training subset. We therefore re-evaluated every subject-scaling checkpoint (2, 4, 5, 8, 10, 15, 16 subjects; DIVER-1-0.1s Small, 2 epochs, constant dataset size) on one common validation split from the full corpus:
| subjects | 2 | 4 | 5 | 8 | 10 | 15 | 16 |
|---|---|---|---|---|---|---|---|
| test loss | 0.260 | 0.242 | 0.208 | 0.207 | 0.203 | 0.210 | 0.226 |
Test loss falls steeply up to ~10 subjects and then flattens and slightly reverses. Because total dataset size is held constant, adding subjects reduces data per subject, so the benefit of subject diversity saturates in this regime. We will add this analysis to the revision appendix today (Scaling Law Experiment Details, "Number of Subjects").
Response to Reviewer Ect3 [3/3]
[Q5] Input to STCPE (L160)
"What is the input to the STCPE computation? It cannot be X as this is itself a function of E_STCPE."
You are right that the notation in L160 is circular as written, and we will fix it in the revision today. The input to STCPE is the content embedding of each token — the sum of the CNN patch embedding, the spectral embedding, and the modality/coordinate embedding — before any positional term is added. Call this X̃. STCPE computes E_STCPE = f(X̃) where f is a pair of lightweight sliding-window transformer modules applied along the time and channel axes — a local aggregation over neighboring tokens, and the transformer input is X̃ + E_STCPE. There is no self-reference: E_STCPE is a function of the content embeddings only. This is the standard conditional-positional-embedding construction (Chu et al., 2023), which is what makes the bias channel-permutation-equivariant.
[Q6] ≤30 temporal patches and the 0.1s model
"Line 230 talks about sampling less than 30 temporal patches. I guess this changes in the 0.1s patch model."
Your reading is correct, and we will state this explicitly in the revision today. The cap is defined in patches, not seconds: both granularities sample up to 30 temporal patches. The absolute context length therefore differs — up to 30 s for DIVER-1-1s and up to 3 s for DIVER-1-0.1s (mean ≈ 22.5 patches ≈ 22.5 s / 2.25 s under Beta(3,1) resampling). This is the intended trade-off of the granularity axis: the 0.1s model gains temporal resolution and gives up context length.
Response to Reviewer Ect3 [1/3]
Thank you for your detailed and thoughtful review, and for your accurate summary of DIVER-1's architecture and pretraining. We are glad you found the scaling results informative, especially that lower pretraining loss does not reliably carry over to downstream performance. We agree this points to the need for better pretraining objectives for multivariate neural time series. In response, we pretrained DIVER-1 on BrainTreebank only, so we could compare it with PopT-v2 on the same data. We also clarified channel-free operation, the cost of full attention, test loss versus subject count, and the STCPE notation. We respond point by point below.
[W1, Q1] Comparison with prior foundation models – PopT-v2 pretrained on identical data
BrainTreebank-only pretraining compared with PopT-v2. We thank the reviewer for requesting a comparison that controls for pretraining data. We pretrained the DIVER-1 architecture (d=256, 12 layers, 0.1-s patches, and the same masked-reconstruction objective) using only BrainTreebank. We matched the subjects and sessions used by PopT-v2, excluded the Neuroprobe test recordings, and applied the same Laplacian preprocessing. We then used the identical iMindBench evaluation protocol. For the frozen comparison, both models used a logistic readout on layer-11 features.
The table reports test AUROC ×100 on the Main units (Neuroprobe: 49; BYD: 254; Pippi: 44). Overall is the equal-weight mean across the three datasets.
Paired, patient-level bootstrap comparisons used 20,000 resamples. The BTB-only DIVER-1 model at 32 epochs scored 0.51 points above frozen PopT-v2 (95% CI [−0.78, +1.91], p=0.47) and 0.75 points below the published fine-tuned PopT-v2 result (95% CI [−1.72, +0.32], p=0.17). Neither difference was statistically significant. The direct frozen-model comparison therefore shows comparable overall performance with approximately 1.1% of PopT-v2's reported pretraining FLOPs (1.8e16 versus 1.6e18). The dataset-level results were mixed: DIVER-1 was higher on Neuroprobe and Pippi and lower on BYD.
Extending BTB-only pretraining did not improve the overall score: 64 versus 32 epochs yielded +0.04 points (95% CI [−0.33, +0.38], p=0.84), while 128 versus 64 epochs yielded −0.34 points (95% CI [−0.71, +0.09], p=0.13). Thus, we did not observe a benefit from additional epochs in this setting. The released DIVER-1 model, pretrained on a larger and more diverse iEEG corpus, scored 1.25 points above the 32-epoch BTB-only model (95% CI [+0.50, +2.11], p<0.001). This result is consistent with a benefit from broader pretraining data, although the present comparison does not isolate dataset size from other differences in the pretraining corpus.
We are also training PopT-v2 on the larger dataset used for DIVER-1 to examine the reverse data-matched comparison. This experiment is computationally demanding and is still in progress. If the results become available during the discussion period, we will report them here. Otherwise, we will include them in the camera-ready version if the paper is accepted.