acceptodds
Under review as a conference paper at ICLR 2027

NEST: A Large-Scale Benchmark for Predicting fMRI in Native Anatomy from EEG and Structural MRI

Abstract

Predicting functional MRI (fMRI) from non-invasive electrophysiology (EEG) could enable cost-effective, high-resolution brain imaging, yet existing benchmarks omit subject-specific anatomies and rely on small cohorts that include fewer than subjects. We propose NEST (Native-anatomy EEG–sMRI–fMRI Tri-modal benchmark), a large-scale benchmark for predicting 4D fMRI in native anatomy from EEG and structural MRI during naturalistic viewing. By aligning separately recorded EEG and fMRI through a shared film timeline, rather than requiring simultaneous recording, NEST scales to and subjects watching two films, over more subjects than the largest public EEG–fMRI dataset. Models are evaluated on unseen subjects by voxel-level reconstruction, regional temporal correlation, and functional connectivity. We also propose NESTOR, which uses modality-specific conditioning, in which the subject's sMRI serves as a static anatomical condition and the EEG as a time-varying condition. NESTOR outperforms existing EEG-to-fMRI models by a clear margin, highlighting the importance of the subject's native anatomy. We demonstrate that NESTOR transfers to unseen movies in a zero-shot setting, indicating that it truly predicts fMRI from EEG signals rather than merely memorizing stimulus-specific fMRI patterns.

Primary area:
applications to neuroscience & cognitive science
Keywords:
fMRI, MRI, EEG, multi-modal, brain, native anatomy

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MKqTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper curates a new dataset for predicting fMRI from EEG, built from existing public data. The same subjects watched the same films in separate EEG and fMRI sessions, and the two recordings are aligned through the shared film timeline. The dataset is larger than any previously published EEG-to-fMRI dataset. Additionally, the authors propose a new EEG-to-fMRI model that uses structural MRI as anatomical guidance, and it outperforms two previous models on their data.

Critical strengths
  • New large-scale dataset, reproducible from public data and available for the community to work on.
  • The proposed model outperforms previous methods on this dataset.
Critical weaknesses
  • The benchmark relies on a strong assumption that brain activity is the same across two different sessions of viewing the same stimulus. Furthermore, it has been demonstrated that responses differ between the first viewing of a stimulus and subsequent viewings. The paper acknowledges that session-specific spontaneous fluctuations cannot be predicted from EEG, but it assumes the stimulus-driven response is identical across sessions and does not address repeated-viewing effects or other variations due to internal state (e.g., attention, arousal, or fatigue). This should be disclosed in the paper as part of its limitations.
  • The main motivation for this line of research is application-driven: providing better tools for analyzing EEG recordings in both research and clinical settings. However, the paper does not demonstrate that the larger dataset or the proposed model is useful for any downstream task (although it motivates the work with clinical applications). This is the central question in this line of work and should be addressed. Furthermore, the setup is somewhat constrained relative to the kinds of experiments done in research and clinical settings, and its generalization is questionable.
  • There are established benchmarks for this task, with multiple works evaluated on them. Why does the paper not report results on any of them? Relying on running other people’s code can be problematic, especially when it is the only comparison provided.

To summarize, understanding the significance of this work requires knowing whether the data and the model generalize beyond the constrained setting of the dataset, and the limitations should be provided and discussed.

Questions for authors

No additional questions beyond what is written in limitations.

7 comments
AuthorsOpenReview ↗

Response to Reviewer MKqT (1/2)

We appreciate the reviewer for the detailed feedback. Below are our responses to the specific points raised.

Re: The benchmark relies on a strong assumption that brain activity is the same across two different sessions of viewing the same stimulus. Furthermore, it has been demonstrated that responses differ between the first viewing of a stimulus and subsequent viewings. The paper acknowledges that session-specific spontaneous fluctuations cannot be predicted from EEG, but it assumes the stimulus-driven response is identical across sessions and does not address repeated-viewing effects or other variations due to internal state (e.g., attention, arousal, or fatigue). This should be disclosed in the paper as part of its limitations.

We agree and will discuss repeated-viewing effects and internal states in the Discussion (Section 6). We note that NEST requires only a shared stimulus-driven component rather than an identical response (Section 3.1).

Aligning separate-session recordings via a shared stimulus is common practice in multimodal neuroscience [R1-4], and prior studies show that the stimulus-driven response remains consistent across repeated viewings, although its strength decreases [R5-7]. Simultaneous EEG-fMRI avoids this assumption but introduces gradient and ballistocardiogram artifacts and requires specialized hardware, which limits cohort sizes [R8-9].

Our results show that NESTOR predicts the fMRI of unseen subjects from EEG recorded in a different session, and also on a film whose fMRI it never saw during training. This suggests the potential of using non-simultaneous recordings that follow standard clinical protocols and are much easier to collect at scale.

Reference

[R1] Cichy, Radoslaw Martin, et al. "Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence." Scientific reports 6.1 (2016): 27755.

[R2] Cichy, Radoslaw M., and Aude Oliva. "AM/EEG-fMRI fusion primer: resolving human brain responses in space and time." Neuron 107.5 (2020): 772-781.

[R3] Haufe, Stefan, et al. "Elucidating relations between fMRI, ECoG, and EEG through a common natural stimulus." NeuroImage 179 (2018): 79-91.

[R4] Hebart, Martin N., et al. "THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior." Elife 12 (2023): e82580.

[R5] Dmochowski, Jacek P., et al. "Correlated components of ongoing EEG point to emotionally laden attention–a possible marker of engagement?." Frontiers in human neuroscience 6 (2012): 112.

[R6] Golland, Yulia, et al. "Extrinsic and intrinsic systems in the posterior cortex of the human brain revealed during natural sensory stimulation." Cerebral cortex 17.4 (2007): 766-777.

[R7] Nastase, Samuel A., et al. "Measuring shared responses across subjects using intersubject correlation." Social cognitive and affective neuroscience 14.6 (2019): 667-685.

[R8] Ebrahimzadeh, Elias, et al. "Simultaneous electroencephalography-functional magnetic resonance imaging for assessment of human brain function." Frontiers in Systems Neuroscience 16 (2022): 934266.

[R9] Telesford, Qawi K., et al. "An open-access dataset of naturalistic viewing using simultaneous EEG-fMRI." Scientific Data 10.1 (2023): 554.

AuthorsOpenReview ↗

Response to Reviewer MKqT (2/2)

Re: The main motivation for this line of research is application-driven: providing better tools for analyzing EEG recordings in both research and clinical settings. However, the paper does not demonstrate that the larger dataset or the proposed model is useful for any downstream task (although it motivates the work with clinical applications). This is the central question in this line of work and should be addressed. Furthermore, the setup is somewhat constrained relative to the kinds of experiments done in research and clinical settings, and its generalization is questionable.

We agree that our paper targets specific applications as noted in the primary area Primary Area: applications to neuroscience & cognitive science. However, we would like to kindly claim that our motivation is not to provide tools for analyzing EEG recordings. Rather, we predict native-space fMRI signals from cheaper modalities, EEG and structural MRI.

NESTOR generalizes to unseen subjects on the same film (Table 2) and on a different film without retraining (Table 3), with test cohorts (77 and 82 subjects) larger than the full cohort of any prior benchmark (Table 1). This shows the potential of predicting fMRI for new subjects and new stimuli from EEG and a single structural scan.

Re: There are established benchmarks for this task, with multiple works evaluated on them. Why does the paper not report results on any of them? Relying on running other people’s code can be problematic, especially when it is the only comparison provided.

We propose a new benchmark for native-space fMRI generation from EEG and structural MRI. As we mentioned in L94, there is no existing benchmark for this task, and NEST is the first one. As a baseline, we also propose NESTOR.

We therefore trained two of the most recent models, UnEBOLT and E2fNet, on NEST using their official codebases and released objectives (Table 7), the same splits as NESTOR. This is the standard protocol for evaluating existing methods on a new benchmark in machine learning.

I thank the authors for adding the suggested discussion of limitations. I acknowledge that incorporating sMRI information is an interesting direction. However, my main concern remains: the generalizability of the approach and the data beyond the very limited experimental setup of the dataset.

The authors state that their objective is predicting fMRI from cheaper modalities. In that case, they should demonstrate why the predicted fMRI is useful in the specific setup they are working on, as it has not been established that the approach generalizes beyond that setup.

What would make this compelling for me is a demonstration that this dataset can be leveraged to improve performance on other datasets with established benchmarks. This is especially important since the data was not collected but aggregated from existing sources, so showing that it is useful for other tasks would make the contribution meaningful.

In addition, comparing other methods on your dataset is problematic, as they were optimized for a different setting, and parameter tuning may improve them significantly.

Given the above, I maintain my score.

Reviewer_TgQCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study introduces NEST, a large-scale benchmark for predicting 4D fMRI in subject-native anatomy using EEG and structural MRI. It aligns separately recorded EEG and fMRI through shared movie timelines, yielding cohorts of over 900 subjects for each of two films. The paper also proposes NESTOR, which uses structural MRI to provide anatomical information and EEG to capture temporal dynamics, and together to reconstruct the voxel level fMRI signals. Evaluation on held-out subjects covers voxel-level reconstruction, regional temporal correlation, and functional connectivity. NESTOR outperforms the evaluated baselines on most metrics and demonstrates zero-shot transfer across movies.

Critical strengths
  • The study takes a meaningful step toward larger-scale EEG-to-fMRI research by aligning separately recorded EEG and fMRI through a shared movie timeline. The cohorts of over 900 subjects per movie substantially expand the scale of evaluation compared with previous study.
  • The study predicts 4D fMRI in each subject’s native anatomy while achieving temporal correlations of approximately 0.29-0.31 on held-out subjects. Recovering temporal variation alongside spatial structure is an important strength, particularly since visually plausible volumes alone do not establish accurate functional predictions.
Critical weaknesses
  • The current experiments do not clearly isolate the sources of the performance improvement. Compared with UnEBOLT, NESTOR uses more EEG channels (129 vs. 26), a longer context window (96 vs. 16 seconds), a different EEG encoder, and additional structural MRI input. Although the paper acknowledges some of these differences, the ablations focus on loss terms.
  • The baseline comparison is limited. Since the paper introduces a benchmark, including additional relevant methods, such as NT-ViT and Spec2VolCAMU-Net, CATD (surface wise, might be tricky to train/eval though since this study is focusing on voxel space), etc, would be helpful and make the evaluation more comprehensive.

Minor:

  • I can get that using subject-specific structural MRI helps preserve individual anatomy and supports prediction in native space. However, one motivation for EEG-to-fMRI synthesis is to reduce reliance on MRI acquisition. Requiring an individual structural scan limits the method’s applicability when MRI is unavailable, or cannot be performed and no usable prior scan exists (or when subjects are not able to take scans, e.g., people with implant). As author acknowledged, using a standard anatomical template would be a great idea, but this alternative is not evaluated. It remains unclear how well the model would perform without subject-specific sMRI, particularly whether its ROI-level temporal correlation and FC recovery would still exceed those of EEG-only baselines.
Questions for authors
  • Have you compared conditioning on subject-specific structural MRI with conditioning on a standard anatomical template, such as MNI? How would this affect voxel-level reconstruction and ROI-level predictions?
  • What motivated the decision not to regress motion-related confounds from the fMRI signals? Although results on a low-motion subset are provided, have you examined how motion-confound regression affects the results?
5 comments
AuthorsOpenReview ↗

Response to Reviewer TgQC (1/2)

We thank the reviewer for the constructive feedback on baseline comparisons and settings.

Re: The current experiments do not clearly isolate the sources of the performance improvement. Compared with UnEBOLT, NESTOR uses more EEG channels (129 vs. 26), a longer context window (96 vs. 16 seconds), a different EEG encoder, and additional structural MRI input. Although the paper acknowledges some of these differences, the ablations focus on loss terms.

We emphasize that we propose the first benchmark and models that predict fMRI in native space from EEG and structural MRI. Consequently, no existing baseline uses both EEG and structural MRI to predict native-space fMRI. As a proxy, we compare against existing baselines that predict fMRI from EEG alone, either as 1D signals for regions of interest (ROIs; UnEBOLT) or as voxel-wise 3D volumes (E2fNet).

To assess the contribution of structural MRI, we replaced each subject's structural MRI with the fixed MNI152 template brain on the Despicable Me dataset. As shown in Table TgQC-1, NESTOR with the MNI template performs worse than NESTOR with subject-specific structural MRI, particularly in voxel-wise reconstruction (MSE, PSNR, SSIM), but still outperforms the baselines on most metrics.

Table TgQC-1. Comparison between NESTOR with and without using structural MRI. We use the MNI 152 template instead of individual structural MRI.

ModelMSE (×10−3)PSNRSSIMTemporal rFC rFC MSE
NESTOR (no sMRI)12.22 ± 0.1519.96 ± 0.020.626 ± 0.0030.327 ± 0.0120.281 ± 0.0210.277 ± 0.021
NESTOR8.07 ± 0.1822.28 ± 0.100.804 ± 0.0010.335 ± 0.0060.306 ± 0.0180.230 ± 0.004

We also evaluate UnEBOLT on NEST (Despicable Me) with the same EEG encoder (REVE) and the same number of channels (129) as NESTOR, as well as NESTOR with a shorter temporal window (16 s), in Table TgQC-2. Although REVE with 129 channels improves the temporal and FC correlations of UnEBOLT (third row), both remain lower than those of NESTOR with either a 16 s or a 96 s window.

Table TgQC-2. Comparison of UnEBOLT and NESTOR in NEST (Despicable Me)

ModelTemporal window (s)EEG EncoderEEG channelsTemporal rFC rFC MSE
UnEBOLT (original)16LaBraM260.065 ± 0.0050.154 ± 0.0320.216 ± 0.019
UnEBOLT16REVE1290.071 ± 0.0160.095 ± 0.0220.246 ± 0.053
UnEBOLT16REVE1290.077 ± 0.0060.219 ± 0.0380.224 ± 0.013
NESTOR16REVE1290.145 ± 0.0090.318 ± 0.0080.235 ± 0.014
NESTOR (ours)96REVE1290.335 ± 0.0060.306 ± 0.0180.230 ± 0.004

Re: The baseline comparison is limited. Since the paper introduces a benchmark, including additional relevant methods, such as NT-ViT and Spec2VolCAMU-Net, CATD (surface wise, might be tricky to train/eval though since this study is focusing on voxel space), etc, would be helpful and make the evaluation more comprehensive.

We evaluated NT-ViT [R1] and Spec2VolCAMU-Net [R2] on NEST (Despicable Me) in Table TgQC-3. Because CATD [R3] is not publicly available, we could not reimplement and evaluate it within the rebuttal period. As shown in Table TgQC-3, NESTOR outperforms both methods on all metrics except FC MSE.

Table TgQC-3. Comparison between NESTOR, NT-VIT and Spec2VolCAMU-Net on NEST (Decipicable Me)

ModelMSE (×10−3)PSNRSSIMTemporal rFC rFC MSE
NT-ViT11.55 ± 0.2220.27 ± 0.050.661 ± 0.0040.019 ± 0.0080.136 ± 0.0360.316 ± 0.004
Spec2VolCAMU-Net11.67 ± 0.0920.17 ± 0.030.677 ± 0.0000.001 ± 0.0010.124 ± 0.0020.300 ± 0.002
NESTOR8.07 ± 0.1822.28 ± 0.100.804 ± 0.0010.306 ± 0.0180.230 ± 0.0040.335 ± 0.006

Reference
[R1] Lanzino, Romeo, et al. "Nt-vit: Neural transcoding vision transformers for eeg-to-fmri synthesis." ECCV Workshop on Synthetic Data for Computer Vision. 2024.
[R2] He, Dongyi, et al. "Spec2VolCAMU-Net: a spectrogram-to-volume model for EEG-to-fMRI reconstruction based on Multi-directional Time–Frequency Convolutional Attention Encoder and Vision-Mamba U-Net." Journal of Neural Engineering 22.5 (2025): 056042.
[R3] Yao, Weiheng, et al. "Catd: Unified representation learning for eeg-to-fmri cross-modal generation." IEEE Transactions on Medical Imaging 44.7 (2025): 2757-2767.

AuthorsOpenReview ↗

Response to Reviewer TgQC (2/2)

Re: [minor] I can get that using subject-specific structural MRI helps preserve individual anatomy and supports prediction in native space. However, one motivation for EEG-to-fMRI synthesis is to reduce reliance on MRI acquisition. Requiring an individual structural scan limits the method’s applicability when MRI is unavailable, or cannot be performed and no usable prior scan exists (or when subjects are not able to take scans, e.g., people with implant). As author acknowledged, using a standard anatomical template would be a great idea, but this alternative is not evaluated. It remains unclear how well the model would perform without subject-specific sMRI, particularly whether its ROI-level temporal correlation and FC recovery would still exceed those of EEG-only baselines.

We agree that a primary motivation of EEG-to-fMRI synthesis is to reduce the burden and cost of MRI acquisition, and our framework shares this goal. Unlike functional MRI, which must be acquired in every experimental session, structural MRI captures static anatomy. It needs to be acquired only once, or retrieved from existing clinical records, to support many subsequent EEG experiments across tasks and time points.

Moreover, as shown in Tables TgQC-1 and TgQC-2, NESTOR with the MNI152 template instead of subject-specific structural MRI outperforms the EEG-only baselines in both temporal correlation (0.327 vs. 0.077) and FC correlation (0.281 vs. 0.219).

I thank the authors for the additional experiments, new analyses, and the time and effort invested in this rebuttal. The new results are helpful and address many of my earlier questions.

However, several concerns remain. Subject-specific sMRI still appears necessary for strong voxel-level reconstruction (replacing it with a fixed MNI template substantially degrades voxel metrics, making the proposed method seem to be worse than other voxel based eeg-only approach (NT-ViT and Spec2VolCAMU-Net)), if I'm reading the tables correctly? So this introduces a practical trade-off when MRI acquisition is unavailable.

Also, unless I missed it, I couldn't find a response to my earlier question about motion-confound regression.

That said, I do see strong promise in this approach, as its ROI-level performance outperforms previous ROI-based baselines by a clear margin (even without subject-specific sMRI). Overall, while I appreciate the substantial effort, I am inclined to maintain my original score for now.

Reviewer_xtF8OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for EEG to fMRI prediction. It uses data from around 1000 subjects, aligning EEG and fMRI recoreded in separate sessions through a shared movie timeline. It also proposes NESTOR and shows the improvements over two baselines.

Critical strengths
  • The proposed benchamark is in larger scale compared to existing works.
  • Using subject-specific structural MRI is well motivated.
  • The cross-movie evaluation shows the generalization ability.
Critical weaknesses
  • The contribution of the EEG is not clear.
  • The comparison with baselines is not fair. The baselines have different inputs with the proposed NESTOR.
Questions for authors

There is no further questions for this paper.

2 comments
AuthorsOpenReview ↗

Resposne to Reviewer xtF8

Re: The comparison with baselines is not fair. The baselines have different inputs with the proposed NESTOR. The contribution of the EEG is not clear.

We emphasize that we propose the first benchmark and models that predict fMRI in native space from EEG and structural MRI. Consequently, no existing baseline uses both EEG and structural MRI to predict native-space fMRI. As a proxy, we compare against existing baselines that predict fMRI from EEG alone, either as 1D signals for regions of interest (ROIs; UnEBOLT) or as voxel-wise 3D volumes (E2fNet).

To assess the contribution of EEG, we replaced each subject’s structural MRI with the fixed MNI152 template brain on the Despicable Me dataset. As shown in Table xTF8, NESTOR with the MNI template still outperforms the baselines on most metrics, although its voxel-wise reconstruction (MSE, PSNR, SSIM) is worse than that of NESTOR with subject-specific structural MRI.

Table xtF8. Performance of NESTOR with subject-specific structural MRI and with the MNI152 brain template. Using structural MRI improves the performance, but even without using structural MRI, NESTOR still outperforms existing EEG-only baselines.

ModelMSE (×10−3)PSNRSSIMTemporal rFC rFC MSE
UnEBOLT---0.065 ± 0.0050.154 ± 0.0320.216 ± 0.019
E2fNet12.39 ± 0.7619.87 ± 0.250.647 ± 0.010-0.020 ± 0.0090.109 ± 0.0610.259 ± 0.060
NESTOR (no sMRI)12.22 ± 0.1519.96 ± 0.020.626 ± 0.0030.327 ± 0.0120.281 ± 0.0210.277 ± 0.021
NESTOR8.07 ± 0.1822.28 ± 0.100.804 ± 0.0010.335 ± 0.0060.306 ± 0.0180.230 ± 0.004

Dear Reviewer,

The discussion period is nearing its end, and we haven’t yet seen a reply from you. We encourage you to respond to the authors’ comments, especially if you have further questions about points that may affect your assessment.

Thank you for taking the time to engage in the discussion.

Best,

Your AC

Comments on the paper

1 comment

We thank the area chairs and reviewers for their careful reading of our submission, and especially for the suggestions on ablation studies and additional baselines, which strengthened the paper.

The reviewers found that NEST substantially expands the scale of EEG-to-fMRI research, with cohorts of over 900 subjects per film built from public data that the community can reproduce (xtF8, TgQC, MKqT). They also noted that using subject-specific structural MRI is well motivated (xtF8), that NESTOR recovers temporal dynamics in each subject's native anatomy rather than only visually plausible volumes (TgQC), and that it generalizes to a film it never saw during training (xtF8, TgQC). Reviewers xtF8 and MKqT found the paper clear and easy to follow.

The main concerns were (1) whether the comparison with baselines is fair given their different inputs, and which factors drive NESTOR's improvement (xtF8, TgQC), (2) the limited set of baselines and the lack of results on other benchmarks (TgQC, MKqT), (3) the reliance on subject-specific structural MRI (TgQC).

In our revision and rebuttal, we made the following changes and added the following experiments. All experiments were reported as the mean and standard deviation from five random seeds.

  • Structural MRI ablation (xtF8, TgQC). We replaced each subject's structural MRI with the MNI152 template. NESTOR with the template still outperforms the EEG-only baselines in temporal correlation (0.327 vs. 0.077) and FC correlation (0.281 vs. 0.219), although its voxel-wise reconstruction is worse than with subject-specific structural MRI (Section 5.2).
  • Fair comparison with UnEBOLT (TgQC). We trained UnEBOLT with NESTOR's EEG encoder (REVE) and all 129 channels, and trained NESTOR with UnEBOLT's 16 s window. Under matched settings, NESTOR still achieves higher temporal correlation (0.145 vs. 0.077) and FC correlation (0.318 vs. 0.219) (Section E.2).
  • Additional baselines (TgQC). We evaluated NT-ViT and Spec2VolCAMU-Net on NEST. NESTOR outperforms both on all metrics except FC MSE (Section 5.2).
  • Comparison on NODDI and Oddball (MKqT). we evaluated NESTOR on NODDI and Oddball, showing that the model achieves state-of-the-art performance on their benchmarks in most metrics, with comparable performance on SSIM in NODDI to the best-performing baseline. (Appendix G)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.