acceptodds
Under review as a conference paper at ICLR 2027

DIVER-1: Scaling Intracranial EEG Foundation Models for Transferable Representations

11 views

Abstract

Intracranial EEG (iEEG) provides direct, millisecond-scale recordings of human neural activity, but reusable representation learning is difficult because electrode layouts, anatomical coverage, referencing schemes, and recording conditions vary across patients and centers. We introduce DIVER-1, a self-supervised iEEG foundation model for variable-input recordings that combines any-variate electrode–time attention, spatio-temporal resampling, input-conditioned positional embeddings, and multi-domain masked reconstruction without assuming a fixed electrode montage. We pretrain two variants, DIVER-1-0.1s and DIVER-1-1s, on 5,310 hours of ECoG and SEEG spanning 352k channel-hours, roughly 54 the BrainTreeBank-based pretraining volume. We evaluate DIVER-1 on two held-out benchmarks: iMINDBench for naturalistic decoding and MAYO for seizure detection. DIVER-1-0.1s achieves the best results on both benchmarks — outperforming all prior foundation models and matching the strongest classical method on MAYO—despite being pretrained exclusively on task-free clinical monitoring recordings from external institutions. DIVER-1-1s achieves comparable performance with an order-of-magnitude smaller token budget, offering a practical option for compute-constrained settings. Finally, we present, to our knowledge, the first controlled, compute-aware scaling study of self-supervised iEEG pretraining, varying data scale, subject count, training duration, and model size up to 1.8B parameters. For masked reconstruction, we find that expanding unique recordings and training sufficiently long are more reliable scaling axes than increasing parameter count alone, highlighting the need for broader datasets and self-supervised objectives better aligned with downstream use. Code is available at [link](https://anonymous.4open.science/r/DIVER-1_update).

Primary area:
applications to neuroscience & cognitive science
Keywords:
Neuroscience, Foundation Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Ect3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Despite rapid advancements in machine learning (natural language, vision), decoding neural activity remains a challenging task. Indeed, small-scale, task-tailored solutions (e.g., simple CNNs) are still hard to beat by large-scale foundational models, despite their large parameter counts and associated compute demand in pretraining. This paper aims to advance such foundational models for decoding intracranial electroencephalogram (iEEG), a modality that promises high signal-to-noise ratio that may open the opportunity for high accuracy decoding. The proposed solution, DIVER-1, features invariance to the number and ordering of the channels as well as the time window, enabled through a novel architectural components and spatio-temporal resampling during self-supervised pretraining. DIVER-1 makes two contributions:

  1. Architecture: For computing the input embeddings, the method integrates multi-view input features (CNN features, multi-spectral embeddings, optionally position embeddings) as well as a novel Spatio-Temporal Conditional Positional Embedding (STCPE), which is a content-dependent positional bias that promises to be channel-permutation-equivariant. These inputs are then fed into an any-variate Transformer block (MOIRAI; Woo et al., 2024), which computes full attention over all electrode-time tokens. One of the advantages of DIVER-1 is its flexibility in handling different input time span and numbers of channels as well as the supporting cases where the channel information (location) is not available. This is partially reflected in pretraining, where a spatio-temporal resampling (STR) samples the spatio-temporal input dimensions during training.
  2. Large-scale pretraining: DIVER-1 is pretrained on ECoG and sEEG datasets with 352k channel-hours from 37 subjects. The dataset consists of a publicly available (AJILE12; Peterson et al., 2022) and a private (25 drug-resistant epilepsy patients) recording . Besides pretraining the base models and evaluating their downstream performance on iMINDBench and seizure detection (MAYO), the paper conducts a scaling study (Kaplan et al., 2020).

Overall, the paper makes the following findings:

  • DIVER-1 slightly outperforms all traditional and foundational baselines on iMINDBench (0.648 vs. 0.645 against strongest PopT-v2) and MAYO (0.974 vs 0.964 against leaderboard reimpl. Olson & Mingle; no channel info available to my knowledge).
  • Scaling experiments find that the training loss can generally be reduced with additional data and compute, but this does not necessarily transfer to improved downstream performance.
Critical strengths

I find the insights from the scaling experiments insightful and informative, particularly the non-faithful transfer between pretraining loss and downstream benchmark performance. Besides asking for more high-quality data, the conclusion mainly asks for new design choices that allow a transfer from pretraining to downstream tasks. While language may not be fully immune against such faithful transfers either, their next token prediction pretraining objective is highly effective and simple. Multivariate time-series do not have such a clear (established) notion of pretraining to downstream transfer.

Critical weaknesses

I'm missing a fair comparison between the proposed DIVER-1 and prior foundational models. Having introduced a new architecture (or at least new architectural components) and a new pretraining pipeline featuring significantly more data (closed-source), it is difficult to attribute the observed gains accordingly. An equal-size PopT-v2 architecture may perform as well as, or ever better than, DIVER-1 when trained with the same data. I guess the most straight-forward comparison would be to train DIVER-1 on Brain TreeBank with a similar procedure as PopT-v2.

Questions for authors
  • How does DIVER-1 compare to PopT-v2 when equalizing the pretraining (dataset and compute) and the number of parameters?
  • DIVER-1 allows to exclude the channel location information at the input. I am not sure where this is demonstrated. Is it MAYO? How much would the performance drop if the channel information were removed in iMINDBench? How does the performance compare to natively channel-free foundational models, such as MVPFormer?
  • Having full spatial-time attention is definitely desirable; however, it also comes with additional costs. Can you quantify the cost compared to other equally-sized EEG foundation models?
  • For the scaling experiments: How does the test loss behave with respect to the number of subjects? I only can see downstream performance in Figure 3b.
  • What is the input to the STCPE computation (Line 160)? It cannot be X as this is itself a function of E_STCPE.
  • Line 230 talks about sampling less than 30 temporal patches. I guess this changes in the 0.1s patch model.
9 comments
AuthorsOpenReview ↗

Response to Reviewer Ect3 [1/3]

Thank you for your detailed and thoughtful review, and for your accurate summary of DIVER-1's architecture and pretraining. We are glad you found the scaling results informative, especially that lower pretraining loss does not reliably carry over to downstream performance. We agree this points to the need for better pretraining objectives for multivariate neural time series. In response, we pretrained DIVER-1 on BrainTreebank only, so we could compare it with PopT-v2 on the same data. We also clarified channel-free operation, the cost of full attention, test loss versus subject count, and the STCPE notation. We respond point by point below.

[W1, Q1] Comparison with prior foundation models – PopT-v2 pretrained on identical data

"How does DIVER-1 compare to PopT-v2 when equalizing the pretraining (dataset and compute) and the number of parameters?"

BrainTreebank-only pretraining compared with PopT-v2. We thank the reviewer for requesting a comparison that controls for pretraining data. We pretrained the DIVER-1 architecture (d=256, 12 layers, 0.1-s patches, and the same masked-reconstruction objective) using only BrainTreebank. We matched the subjects and sessions used by PopT-v2, excluded the Neuroprobe test recordings, and applied the same Laplacian preprocessing. We then used the identical iMindBench evaluation protocol. For the frozen comparison, both models used a logistic readout on layer-11 features.

The table reports test AUROC ×100 on the Main units (Neuroprobe: 49; BYD: 254; Pippi: 44). Overall is the equal-weight mean across the three datasets.

ModelPretraining dataTrain FLOPsNeuroprobeBYDPippiOverall
PopT-v2, frozenBrainTreebank1.6e1869.4063.4556.1463.00
PopT-v2, fine-tuned (published)BrainTreebank1.6e1872.2963.1657.3364.26
DIVER-1 (BTB-only, 32 epochs), frozenBrainTreebank1.8e1672.0461.5156.9863.51
DIVER-1 (BTB-only, 64 epochs), frozenBrainTreebank3.7e1672.0361.8456.7863.55
DIVER-1 (BTB-only, 128 epochs), frozenBrainTreebank7.4e1672.0161.2256.3963.21
DIVER-1 (released), frozen5,310 h iEEG—73.0963.0258.1764.76

Paired, patient-level bootstrap comparisons used 20,000 resamples. The BTB-only DIVER-1 model at 32 epochs scored 0.51 points above frozen PopT-v2 (95% CI [−0.78, +1.91], p=0.47) and 0.75 points below the published fine-tuned PopT-v2 result (95% CI [−1.72, +0.32], p=0.17). Neither difference was statistically significant. The direct frozen-model comparison therefore shows comparable overall performance with approximately 1.1% of PopT-v2's reported pretraining FLOPs (1.8e16 versus 1.6e18). The dataset-level results were mixed: DIVER-1 was higher on Neuroprobe and Pippi and lower on BYD.

Extending BTB-only pretraining did not improve the overall score: 64 versus 32 epochs yielded +0.04 points (95% CI [−0.33, +0.38], p=0.84), while 128 versus 64 epochs yielded −0.34 points (95% CI [−0.71, +0.09], p=0.13). Thus, we did not observe a benefit from additional epochs in this setting. The released DIVER-1 model, pretrained on a larger and more diverse iEEG corpus, scored 1.25 points above the 32-epoch BTB-only model (95% CI [+0.50, +2.11], p<0.001). This result is consistent with a benefit from broader pretraining data, although the present comparison does not isolate dataset size from other differences in the pretraining corpus.

We are also training PopT-v2 on the larger dataset used for DIVER-1 to examine the reverse data-matched comparison. This experiment is computationally demanding and is still in progress. If the results become available during the discussion period, we will report them here. Otherwise, we will include them in the camera-ready version if the paper is accepted.

AuthorsOpenReview ↗

Response to Reviewer Ect3 [2/3]

[Q2] Channel-free operation

"I am not sure where this is demonstrated. Is it MAYO? How much would the performance drop if the channel information were removed in iMINDBench? How does the performance compare to natively channel-free foundational models, such as MVPFormer?"

Yes — MAYO is the demonstration: the dataset provides no electrode locations, so DIVER-1 runs fully channel-free there. Our new clinical evaluation (revision Appendix D.2, uploaded today) is a second one: no model receives electrode coordinates, and DIVER-1 is the best frozen encoder on both tasks.

On the performance drop: Table 1 contains the strongest version of this test — models pretrained entirely without the channel 3D-position or sub-modality embeddings lose only 0.6–0.8 AUROC points on iMINDBench.

On MVPFormer: the iMINDBench authors examined MVPFormer and did not include it in the benchmark, because its released configuration encodes each channel in five-second segments with 25 segments of temporal context, which does not map onto the benchmark's one-second decoding windows without a validated adaptation — retaining the released context provides information beyond the benchmark window, while rescaling the input changes the signal presented to the pretrained encoder (Chau et al., 2026, Appendix I.1). The same constraint applies to any comparison we could run: a fair MVPFormer evaluation on iMINDBench is not currently defined. We therefore demonstrate channel-free operation directly, as above.

[Q3] Cost of full spatio-temporal attention

"Can you quantify the cost compared to other equally-sized EEG foundation models?"

Absolute training cost is already reported in Table 2 (total FLOPs per epoch for every model size; e.g., DIVER-1-1s Small, 51M params: 256.4 PFLOPs/epoch — consistent with the DeepSpeed profiler-measured 67.2 GMACs forward per sample × 636,480 samples × 3 for the backward pass). What Table 2 does not break out is the cost of full spatio-temporal attention specifically. At the STR-capped grid (32 electrodes × 30 patches = 960 tokens), the quadratic attention interactions — the only part a factorized design removes — contribute ≈ 17% of forward compute (11.3 of 67.2 GMACs for Small); the Q/K/V/O projections, which a factorized design retains, add a further ≈ 18%. A factorized spatial+temporal design on the same grid would need ≈ 0.7 GMACs for its interactions, so full attention makes DIVER-1 ≈ 1.2× the cost of an equally-sized factorized model. This overhead is bounded by the STR token cap and grows only with longer windows or more electrodes.

[Q4] Test loss vs number of subjects

"How does the test loss behave with respect to the number of subjects? I only can see downstream performance in Figure 3b."

Agreed — this is worth reporting. Test losses across subject subsets were not directly comparable in our original setup because each run was validated on a split of its own training subset. We therefore re-evaluated every subject-scaling checkpoint (2, 4, 5, 8, 10, 15, 16 subjects; DIVER-1-0.1s Small, 2 epochs, constant dataset size) on one common validation split from the full corpus:

subjects2458101516
test loss0.2600.2420.2080.2070.2030.2100.226

Test loss falls steeply up to ~10 subjects and then flattens and slightly reverses. Because total dataset size is held constant, adding subjects reduces data per subject, so the benefit of subject diversity saturates in this regime. We will add this analysis to the revision appendix today (Scaling Law Experiment Details, "Number of Subjects").

AuthorsOpenReview ↗

Response to Reviewer Ect3 [3/3]

[Q5] Input to STCPE (L160)

"What is the input to the STCPE computation? It cannot be X as this is itself a function of E_STCPE."

You are right that the notation in L160 is circular as written, and we will fix it in the revision today. The input to STCPE is the content embedding of each token — the sum of the CNN patch embedding, the spectral embedding, and the modality/coordinate embedding — before any positional term is added. Call this X̃. STCPE computes E_STCPE = f(X̃) where f is a pair of lightweight sliding-window transformer modules applied along the time and channel axes — a local aggregation over neighboring tokens, and the transformer input is X̃ + E_STCPE. There is no self-reference: E_STCPE is a function of the content embeddings only. This is the standard conditional-positional-embedding construction (Chu et al., 2023), which is what makes the bias channel-permutation-equivariant.

[Q6] ≤30 temporal patches and the 0.1s model

"Line 230 talks about sampling less than 30 temporal patches. I guess this changes in the 0.1s patch model."

Your reading is correct, and we will state this explicitly in the revision today. The cap is defined in patches, not seconds: both granularities sample up to 30 temporal patches. The absolute context length therefore differs — up to 30 s for DIVER-1-1s and up to 3 s for DIVER-1-0.1s (mean ≈ 22.5 patches ≈ 22.5 s / 2.25 s under Beta(3,1) resampling). This is the intended trade-off of the granularity axis: the 0.1s model gains temporal resolution and gives up context length.

Reviewer_3udYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an Intracranial EEG Foundation Model, Diver-1. Diver-1 is trained via self-supervised learning by reconstructing masked waveforms, FFTs, and STFTs (for DIVER-1-1s only) from an iEEG cohort of 37 subjects. The model demonstrates transfer to external downstream datasets, including iMINDBench for naturalistic decoding and MAYO for seizure detection.

Furthermore, the paper presents a controlled, compute-aware scaling study of self-supervised iEEG pretraining. It examines how the amount of pretraining data, number of patients, model size, and training duration affect pretraining loss and downstream performance. The authors find that lower reconstruction loss does not consistently translate into better downstream performance, while increasing data and training duration provides more reliable gains than increasing model size. They argue that current iEEG pretraining is in a data-constrained regime. Based on these results and their fitted scaling laws, they recommend prioritizing additional unique recordings, followed by longer training, before increasing model size.

Critical strengths

DIVER-1 achieves competitive performance on iMINDBench and MAYO. The results on datasets separate from pretraining, including the frozen-encoder results on iMINDBench, provide evidence that the learned representations are useful for the evaluated tasks.

Critical weaknesses
  1. Limited downstream evaluation. The authors claim that their scaling study provides a “practical recipe for future iEEG foundation models” (L132–135). However, the downstream evaluation covers naturalistic decoding on three movie-watching datasets, with 5, 29, and 5 subject/session units, respectively, and seizure detection on 8 human participants. Although iMINDBench includes 15 binary targets, these targets are all evaluated in movie-watching settings. Moreover, the downstream scaling analysis in Figure 3(b) is limited to iMINDBench (L397–401). Other clinically meaningful iEEG applications, including event classification, SOZ identification, and anatomical location classification[1], should also be considered in the downstream evaluation. Drawing broad conclusions about iEEG scaling from such limited downstream evaluation is insufficient to support the claim that these findings can guide the training of iEEG foundation models more generally.

  2. The compute-optimal analysis does not directly support the practical training recommendation. Figure 4 identifies compute-optimal configurations based on reconstruction loss (L450–453), and the authors suggest that the fitted law can guide model-size and training-duration choices (L459–460). However, the paper itself finds that reconstruction loss does not reliably predict downstream performance (L418–419). A configuration that minimizes reconstruction loss under a fixed compute budget therefore need not be the best choice for downstream tasks. Figure 3(b) provides relevant downstream trends, but it does not establish that the configurations recommended by Figure 4 improve downstream performance under matched compute budgets. This leaves a logical gap between the optimization target and the practical recommendation.

  3. Limited cohort for population-related claims. The paper links the benefits of additional recordings to broader population coverage (L410–412). However, the 5310-hour pretraining data (it is quite long in recording hours) only contains 37 patients with epilepsy from two data sources. The patient-count experiments involve 2–16 patients from this same pool (L1198–1200). These experiments examine patient diversity within a small cohort, but do not establish how broader population coverage affects scaling. The population-related conclusions should therefore be limited to the cohorts and sampling conditions evaluated.

[1] Duan, Chenda, et al. "Omni-iEEG: A large-scale, comprehensive iEEG dataset and benchmark for epilepsy research." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors
  1. Can the authors evaluate DIVER-1 on more clinically meaningful iEEG tasks beyond movie watching?

  2. Does the reconstruction objective mainly favor low-frequency components? Although the FFT and STFT targets are log-transformed, the objective also includes raw-waveform MSE. What proportion of the total loss and its reduction comes from each frequency band? In particular, does a lower loss also improve the reconstruction of transient and higher-frequency features relevant to tasks?

  3. How much of the downstream performance comes from pretraining? On iMINDBench, a logistic regression on Multi-STFT features already reaches 0.628 Overall AUROC, while frozen DIVER-1-0.1s reaches 0.648 and frozen DIVER-1-1s reaches only 0.633 (Table 9, L1410–1416). On BYD, frozen DIVER-1-1s (0.616) is even lower than the STFT logistic regression (0.624). However, the paper does not include a randomly initialized or trained-from-scratch model with the same architecture. How does DIVER-1 perform with random initialization under the same frozen-probe protocol, and when trained from scratch on the downstream tasks?

  4. The paper does not describe how the compute-optimal allocation in Figures 4(a) and 8(a) is derived from the fitted scaling law. It would be helpful if the authors provide the work through for the optimal model and data sizes as compute increases, the fitted parameters used, and how the computation led to the fold increases (L1280–1281) and finally lead to the conclusion "scale data before parameters" (L431)?

9 comments
AuthorsOpenReview ↗

Response to Reviewer 3udY [1/4]

Thank you for your thorough and constructive review. We appreciate that you found DIVER-1 competitive on iMINDBench and MAYO, and that the frozen-encoder results on datasets separate from pretraining show the learned representations are useful. Your comments led to several new analyses in the revision: a clinical evaluation of SOZ and resected-channel localization on 56 epilepsy-surgery patients, a matched-compute downstream test of the compute-optimal allocation, a frequency-band breakdown of the reconstruction loss, random-initialization and from-scratch baselines, and the full scaling-law derivation. We respond point by point below.

[W1, Q1] Adding more clinically meaningful iEEG tasks – Omni-iEEG

"Can the authors evaluate DIVER-1 on more clinically meaningful iEEG tasks beyond movie watching?"

We first audited Omni-iEEG. In three of its five tasks, the labels can be predicted from which site or recording a window came from, without looking at the neural signal. A model can therefore score well on these tasks without decoding anything clinically relevant, so they do not reliably measure what an iEEG model has learned. We therefore evaluate on the original HUP epilepsy-surgery release (OpenNeuro ds004100), using a protocol that removes these shortcuts (details below).

Clinical evaluation (56 patients; frozen encoders; no electrode coordinates; one leave-patient-out linear probe; AUROC computed within each patient and averaged, so neither patient nor site identity can help):

encoderSOZ channelsresected channels
DIVER-1.723.711
BrainBERT.711.679
BaRISTA.674.659
PopT-v2.674.653
untrained encoder.689.666

DIVER-1 is the best frozen encoder on both tasks — significantly ahead of PopT-v2 on both (paired patient bootstrap, Holm-corrected: +.049 and +.057, p = .001) and of BaRISTA on resection. Window-level seizure detection is near ceiling for every model (an untrained encoder reaches ≈ .90) and does not separate them.

On Omni-iEEG: its split shares recording sites between train and test while several labels exist at only one site (all ictal recordings are HUP; all awake recordings are SourceSink). A baseline that sees only the site identity — never the signal — reaches AUROC 0.76 on soz_outcome and 0.77 on sleep_awake, beating three of six published models on soz_outcome. "Ictal" is labelled per recording: 53% of ictal windows contain under 5 s of seizure, and a recording-identity control reaches 0.976 on ictal-vs-interictal. The anatomical and HFO tasks pass our checks. We will add the full audit, protocol, and results to the revised appendix (D.2–D.3) today.

AuthorsOpenReview ↗

Response to Reviewer 3udY [2/4]

[W2] Compute-optimal training recommendations

"The compute-optimal analysis does not directly support the practical training recommendation."

You are right that Figure 4 optimizes reconstruction loss, and we ourselves report that this loss does not reliably predict downstream performance. We will revise L459–460 today to scope the fitted law to pretraining loss.

We still think the analysis has practical value. Most groups cannot afford a data-constrained scaling study to choose model size and training duration, so a loss-derived allocation rule will in practice be used as a default. The question is therefore not whether the rule finds the downstream-optimal configuration — it cannot, since loss does not predict downstream performance — but whether following it reliably steers compute into the region where downstream performance is near its best. Our claim is scoped to a standard setting: masked-reconstruction pretraining of a Transformer, evaluated on naturalistic decoding.

What we ran. The paper's L=12 sweeps varied one factor at a time, and training the full width × epoch grid at L=12 was not feasible within the compute and time available during the discussion period, so we repeated the analysis at depth 2: a width × epoch grid (widths 256–1536, i.e. 2–80M params; epochs 1–32; 54 runs over both granularities), each evaluated by linear probing on iMINDBench-BYD (15 tasks). We refit the data-constrained law on this grid, and also trained, beyond the grid, the model the fitted law recommends for a larger compute budget (0.1s: 34.9M params × 36 epochs; 1s: 31.7M × 42), to check whether following the law's own recommendation lands near the best downstream performance. Depth 2 is a reasonable proxy: for 0.1s it reaches similar loss to L=12 at 2 epochs (L=2 vs L=12 — d256: .0950 vs .0956; d512: .0935 vs .0910) and reproduces the paper's exponents (α = 0.52 vs 0.525, β = 0.13 vs 0.125; R² = 0.92); for 1s the fit is equally tight (R² = 0.93) but more parameter-leaning (α = 0.66 vs 0.38, β = 0.15 vs 0.35), while keeping the property the recommendation rests on — repeated data saturates later than excess parameters (RD∗R^*_D = 10.6 vs RN∗R^*_N = 6.4; paper: 9.5 vs 3.4).

Result 1 — at matched compute, the rule matches or beats spending compute on parameters (mean AUROC over 15 tasks):

GranularityCompute (params × epochs)Recommended: smaller × longer (params × epochs → AUROC)Not recommended: larger × shorter (params × epochs → AUROC)
0.1s≈ 70M2.2M × 32 → .566; 8.8M × 8 → .56034.9M × 2 → .534; 78.3M × 1 → .540
≈ 150M8.8M × 16 → .555; 19.7M × 8 → .55834.9M × 4 → .550; 78.3M × 2 → .545
≈ 300M8.8M × 32 → .559; 19.7M × 16 → .55734.9M × 8 → .541; 78.3M × 4 → .549
≈ 630M19.7M × 32 → .55878.3M × 8 → .557
1s≈ 75M2.5M × 32 → .547; 9.4M × 8 → .55536.2M × 2 → .547; 80.3M × 1 → .550
≈ 155M9.4M × 16 → .559; 20.6M × 8 → .56036.2M × 4 → .550; 80.3M × 2 → .546
≈ 310M9.4M × 32 → .565; 20.6M × 16 → .56436.2M × 8 → .554; 80.3M × 4 → .556
≈ 650M20.6M × 32 → .57180.3M × 8 → .561

The recommended configuration wins 13/13 head-to-head pairs (0.1s) and 11/13 (1s; 1 tie, 1 loss), by +.012 / +.009 AUROC on average. Across the grid, adding parameters gives no downstream gain (corr(log N, AUROC) = −0.14 / +0.18) while longer training helps (corr(log epochs, AUROC) = +0.78 / +0.55) — even for 1s, where the loss fit leans toward parameters.

Result 2 — the configurations the law selects land near the downstream best, but not beyond it. The configurations the law recommends — 19.7M/20.6M × 32 epochs within the grid and the larger model trained beyond it (0.1s: .558 / .570; 1s: .571 / .568) — all lie within .012 AUROC of the best configuration we trained (.570 / .571), i.e. within run-to-run noise. The larger model reaches the lowest (minimum) validation loss of all runs (.0721 / .0821) without a further downstream gain.

So the rule does not guarantee the downstream optimum, but at a fixed budget it reliably steers compute into the near-best region and away from the clearly worse large-and-brief configurations. We will add the grid, the matched-compute comparison, and the fits to the appendix today, and state the size-axis result as "no detectable gain" rather than measured harm.

AuthorsOpenReview ↗

Response to Reviewer 3udY [3/4]

[W3] Population and cohort limitations

"The population-related conclusions should therefore be limited to the cohorts and sampling conditions evaluated."

Agreed. Our cohort is 37 epilepsy patients from two sources, and the subject-count experiments draw 2–16 patients from this same pool, so they measure patient diversity within this cohort, not broader population coverage. We will scope the population-related statements (L410–412, Section 4.2) accordingly in the revision today. The new fixed-validation subject-scaling analysis (added to the revision today) is consistent with this scoping: under a constant data budget, the benefit of adding subjects saturates near 10 subjects.

[Q2] Proportion of the total loss and reduction from each frequency band

"Does the reconstruction objective mainly favor low-frequency components? … What proportion of the total loss and its reduction comes from each frequency band?"

We thank the reviewer for this question. To assess whether our model disproportionately emphasizes low-frequency activity, we analyzed the frequency-specific reconstruction loss and conducted a controlled pretraining ablation using full-band, <30-Hz, and ≥30-Hz signals.

1. Contribution of each frequency band to the reconstruction loss and its reduction.

Our objective equally weights waveform MSE and log1p-compressed spectral-amplitude MSE, which contribute 49% and 51% of the final held-out loss. We attributed the spectral term by frequency bin and the waveform term using a Parseval decomposition of the residual. Frequencies below 30 Hz account for 58% of the spectral loss, despite comprising only 15% of its bins. They account for 86% of the waveform loss and 80% of signal power. Combining both terms, 72% of the final loss comes from <30 Hz and 28% from ≥30 Hz.

We evaluated saved checkpoints on fixed held-out segments, masks, and crops. From random initialization, broadband high-frequency error accounts for 64% of the total reduction. We therefore also compare epoch 1 with the final checkpoint. Total loss fell by 20.9% over this interval, with 58% of the reduction from <30 Hz and 42% from ≥30 Hz. Relative to epoch 1, the ≥30-Hz loss fell more than the <30-Hz loss (28.4% vs. 17.5%).

High-frequency reconstruction improved as loss decreased. Band-normalized spectral error fell from 0.414 to 0.308 for ≥30 Hz and from 0.372 to 0.322 for <30 Hz. Waveform residual error relative to band power fell from 0.278 to 0.177 and from 0.357 to 0.285, respectively. The full-band model reconstructed ≥30-Hz spectral content at least as accurately as the model pretrained only on that band (normalized MSE 0.31 vs. 0.33).

2. Full-band pretraining yields the best downstream performance.

We pretrained three models using identical architectures, datasets, and training schedules (8 epochs), varying only the pretraining frequency band. Each model was evaluated on full-band, <30-Hz, and ≥30-Hz downstream inputs using a frozen encoder and linear readout.

Overall AUROC (×100), iMINDBench Main split

PretrainingFull<30 Hz≥30 Hz
Full62.9062.5459.12
<30 Hz60.6461.8656.19
≥30 Hz60.7160.5257.68

Full-band pretraining outperforms frequency-restricted pretraining, including when downstream input matches the pretraining band. Removing ≥30-Hz information reduces overall AUROC by only 0.36 points, indicating that lower frequencies carry much of the task-relevant information. Yet ≥30-Hz input alone achieves 59.12, above chance (50).

3. High-frequency contributions depend on the downstream task.

For acoustic and transient-related tasks, ≥30-Hz input retains much of the full-band performance, including speech decoding (Neuroprobe 84.05 vs. 85.54; BYD 71.00 vs. 73.88) and pitch decoding (BYD 58.83 vs. 58.90). Adding ≥30-Hz information improves speech decoding by 2.26 points on Neuroprobe and 1.14 points on BYD, and sound-onset decoding by 0.92 points on Neuroprobe. In contrast, ≥30-Hz-only input substantially reduces performance for word index (−18.1), Δvolume (−11.6), and word gap (−8.2) on Neuroprobe.

Overall, low frequencies dominate the absolute loss, but ≥30-Hz activity accounts for a substantial share of its reduction after epoch 1. High-frequency reconstruction improves with training and provides task-specific downstream benefits. Full-band pretraining yields the strongest downstream representations.

We used a 30-Hz split because of compute and time constraints; finer frequency decomposition is left for future work.

Reviewer_j9G4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that existing iEEG FMs face 3 botlenecks, which are limited psatiotemporal interactions, variable electrode configurations, and the lack of scaling law studies. To addresses these problems, the authors propose DIVER-1 combining any-variate attention, conditional positional embeddings, spatiotemporal resmpling and multi-domain reconstruction. The model was pretrained on 5310 hours of iEEG and inverstigate how data size, model size and training duration affect the representation. The model is evaluated on iMINDBench and MAYO to show the improvements over previous iEEG foundation models.

Critical strengths
  1. This paper addresses very relevant research gaps, especially the lack of systematic understanding of scaling laws for iEEG FMs.
  2. The authors are careful with the experimental design, using cross-dataset and leakage-aware evaluation. Table 1 also shows that their design choices lead to real improvements, especially spatio-temporal resampling.
  3. The scaling study shows that increasing model size decreases pretraining loss but does not necessarily improve downstream performance.
  4. The authors also observe that data volume and sufficient training duration matter more than simply increasing the parameter count.
  5. The findings in Points 3 and 4 might sound intuitive, but as far as I know, no previous work has investigated these questions at this scale and level of detail for self-supervised iEEG pretraining. I think these experiments provide valuable insights and could have a meaningful impact on future research in this field.
Critical weaknesses
  1. The authors should provide more ablation studies to show to what extent each proposed bottlenech is actually solved. For example, Tab. 1 shows only marginal improvements from attention and positional components.
  2. I suggest the authors should update the figures. Fig. 1 contains too many small texts, while Fig. 2-4 could be benefitted from larger fonts and clearer legends. You should make them more visually appealing and easier to read.
Questions for authors

See Weaknesses

3 comments

Thank you for your careful and encouraging review. We are glad you found the scaling study useful. We address your two concerns below.

[W1] Ablations per bottleneck

"The authors should provide more ablation studies to show to what extent each proposed bottlenech is actually solved. For example, Tab. 1 shows only marginal improvements from attention and positional components."

Each of the three bottlenecks maps to specific components, and Table 1 ablates them individually (all at d_model = 256, 8 epochs, frozen features; full model Overall = 64.8):

BottleneckComponent removedOverall AUROCΔ
—full DIVER-1-0.1s64.8
(1) spatio-temporal interactionsany-variate attention63.6−1.2
RoPE64.1−0.7
both (vanilla attention)62.6−2.2
(2) variable electrode configurationsSTCPE63.5−1.3
channel + time resampling62.0−2.8
(3) scaling—addressed in §4

Yes, each single-component delta for attention and positional terms is small (−0.7 to −1.3 AUROC points). Three facts push back on reading this as "the components barely matter." (1) The effects add up: removing RoPE and any-variate attention together costs −2.2, more than the two alone. (2) The drop shows up in all three datasets (NP, BYD, Pippi), so it is not noise from one benchmark. (3) Accuracy on a fixed electrode layout is not what these components are for. STCPE and any-variate attention are what let DIVER-1 accept any number of electrodes in any order — the channel-free MAYO setting cannot run at all without them. A benchmark with one fixed layout therefore understates their value.

[W2] Figures

"I suggest the authors should update the figures. Fig. 1 contains too many small texts, while Fig. 2-4 could be benefitted from larger fonts and clearer legends."

Thank you for this suggestion. We agree, and we are revising Fig. 1 (less in-figure text, larger minimum font) and Figs. 2–4 (larger fonts, clearer legends). We will upload the revised PDF with the updated figures within the next day and will note it here once it is posted.

Dear Reviewer j9G4,

Thank you again for your review. As the discussion period closes in about a day, we would be grateful to hear whether our responses address your concerns, and are happy to clarify anything further.

Best regards,

The authors

Dear Reviewer j9G4,

As promised, the revised PDF is now on OpenReview. Figure 1 is redrawn with less in-figure text and a larger minimum font, and Figures 2 and 3 use larger axis and tick labels. Figure 4 has also changed: following Reviewer 3udY's point about the compute-optimal allocation, the data–model allocation panel has been removed and the allocation is now reported numerically in Table 10, so Figure 4 shows the IsoLoss contours alone. Table 1, which your ablation question concerns, is unchanged.

All revisions are marked in colour in the PDF. We would be glad to hear whether the figures now read more clearly, and to make further adjustments.

Best regards, The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of the Discussion Period

Thank you to the reviewers for two rounds of discussion. We have answered all points, and the revised PDF is on OpenReview with the changes marked in colour (§3, §4.1–4.3, App. C.1, C.3, D.1–D.5): blue for what the revision adds, purple for the corrections made after your follow-up comments.

Where the gains come from: all three reviewers raised this, prompting three new pretraining runs. DIVER-1 trained on BrainTreebank alone matches PopT-v2 (63.51 vs 63.00 Overall, p = 0.47) at 1.1% of its FLOPs, and PopT-v2 retrained on our corpus at a matched 2.3e17 FLOPs falls 3.34 points below DIVER-1 when frozen (p < 0.001); both runs are reported in the reviewer threads. Random-init and from-scratch controls are in the revised PDF (Table 13), and put the contribution of pretraining at +3.07 / +2.53 frozen and +4.17 / +3.44 fine-tuned.

On the scaling recipe. The compute-optimal allocation is now computed from the fitted Eq. (3) exponents and compared against a language-domain fit in the same Chinchilla form, so both sides come from one procedure (Table 10: p_D = 0.518 at 1 s and 0.808 at 0.1 s, against 0.487). At 1 s the margin is within the standard errors on the language exponents, so we report the shares numerically rather than as a domain contrast; Figures 4 and 8 now present the compute-optimal frontier. A new 54-run width × epoch grid tests the recommendation downstream at matched compute, where it wins 13 of 13 iso-compute pairs at 0.1 s and 11 of 13 at 1 s (Tables 25–26). We also scoped the claims: the fit speaks to pretraining loss, the grid conclusions to DIVER-1-0.1s, and "unique data first" is motivated rather than directly tested. The measurements are unchanged.

On evaluation breadth, we added a coordinate-free clinical evaluation on HUP epilepsy-surgery iEEG (56 patients, leave-patient-out, within-patient scoring), where DIVER-1 is the best encoder for seizure-onset-zone and resected-channel identification — significantly ahead of PopT-v2 (+.049 / +.057, p = .001), but its lead over an untrained encoder does not survive correction. We state this explicitly (App. D.2). The audit behind our choice of HUP over Omni-iEEG is in App. D.3. In response to Reviewer 3udY's question, we also showed that the recording center is recoverable from the signal alone (AUROC .90–.94, leave-patient-out across two hospitals), which is why a split that is not site-disjoint cannot separate clinical state from center recognition. We further showed channel-free operation on the released checkpoint (−1.13 points, still above the Multi-STFT baselines), ran MVPFormer on MAYO, and re-analysed the objective with bands that isolate ripples after Reviewer 3udY noted that a 30-Hz split does not.

Deferred to the camera-ready

The token-matched grid that would test "unique data first"; a capacity-versus-diversity grid at a fixed subject count; a leave-one-site-out multi-center clinical evaluation; and fast-ripple analysis, which needs a corpus sampled above 500 Hz.

Where the revisions are

LocationChangeRaised by
§3STCPE input notation corrected; patch cap stated in patches, with context length per granularityEct3 Q5, Q6
§4.2Population claims scoped to the evaluated cohort3udY W3
§4.3Recipe stated as a measured ordering; compute-optimal fit scoped to pretraining loss; corrected shares; "unique data first" marked as motivated3udY W2, Q4
Figs. 4, 8Compute-Optimal Data–Model Tradeoff panels removed; allocation moved to Table 103udY Q4
App. C.1, Table 8Subject-disjoint held-out loss versus number of subjectsEct3 Q4
App. C.3, Tables 9–10Corrected derivation; compute-optimal shares against a matched language-domain fit3udY Q4
App. D.1, Table 13Random-initialised and from-scratch controls3udY Q3
App. D.2, Tables 14–15Coordinate-free clinical evaluation on HUP3udY W1, Q1
App. D.3, Tables 16–24Audit of the Omni-iEEG benchmark3udY W1
App. D.4, Tables 25–26Matched-compute width × epoch grid, scoped to DIVER-1-0.1s3udY W2
App. D.5, Table 27Frequency-band contribution to the reconstruction objective3udY Q2
Figs. 1–4Larger fonts, clearer legendsj9G4 W2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.