- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper curates a new dataset for predicting fMRI from EEG, built from existing public data. The same subjects watched the same films in separate EEG and fMRI sessions, and the two recordings are aligned through the shared film timeline. The dataset is larger than any previously published EEG-to-fMRI dataset. Additionally, the authors propose a new EEG-to-fMRI model that uses structural MRI as anatomical guidance, and it outperforms two previous models on their data.
Critical strengths
- New large-scale dataset, reproducible from public data and available for the community to work on.
- The proposed model outperforms previous methods on this dataset.
Critical weaknesses
- The benchmark relies on a strong assumption that brain activity is the same across two different sessions of viewing the same stimulus. Furthermore, it has been demonstrated that responses differ between the first viewing of a stimulus and subsequent viewings. The paper acknowledges that session-specific spontaneous fluctuations cannot be predicted from EEG, but it assumes the stimulus-driven response is identical across sessions and does not address repeated-viewing effects or other variations due to internal state (e.g., attention, arousal, or fatigue). This should be disclosed in the paper as part of its limitations.
- The main motivation for this line of research is application-driven: providing better tools for analyzing EEG recordings in both research and clinical settings. However, the paper does not demonstrate that the larger dataset or the proposed model is useful for any downstream task (although it motivates the work with clinical applications). This is the central question in this line of work and should be addressed. Furthermore, the setup is somewhat constrained relative to the kinds of experiments done in research and clinical settings, and its generalization is questionable.
- There are established benchmarks for this task, with multiple works evaluated on them. Why does the paper not report results on any of them? Relying on running other people’s code can be problematic, especially when it is the only comparison provided.
To summarize, understanding the significance of this work requires knowing whether the data and the model generalize beyond the constrained setting of the dataset, and the limitations should be provided and discussed.
Questions for authors
No additional questions beyond what is written in limitations.
7 comments
Response to Reviewer MKqT (2/2)
Re: The main motivation for this line of research is application-driven: providing better tools for analyzing EEG recordings in both research and clinical settings. However, the paper does not demonstrate that the larger dataset or the proposed model is useful for any downstream task (although it motivates the work with clinical applications). This is the central question in this line of work and should be addressed. Furthermore, the setup is somewhat constrained relative to the kinds of experiments done in research and clinical settings, and its generalization is questionable.
We agree that our paper targets specific applications as noted in the primary area Primary Area: applications to neuroscience & cognitive science. However, we would like to kindly claim that our motivation is not to provide tools for analyzing EEG recordings. Rather, we predict native-space fMRI signals from cheaper modalities, EEG and structural MRI.
NESTOR generalizes to unseen subjects on the same film (Table 2) and on a different film without retraining (Table 3), with test cohorts (77 and 82 subjects) larger than the full cohort of any prior benchmark (Table 1). This shows the potential of predicting fMRI for new subjects and new stimuli from EEG and a single structural scan.
Re: There are established benchmarks for this task, with multiple works evaluated on them. Why does the paper not report results on any of them? Relying on running other people’s code can be problematic, especially when it is the only comparison provided.
We propose a new benchmark for native-space fMRI generation from EEG and structural MRI. As we mentioned in L94, there is no existing benchmark for this task, and NEST is the first one. As a baseline, we also propose NESTOR.
We therefore trained two of the most recent models, UnEBOLT and E2fNet, on NEST using their official codebases and released objectives (Table 7), the same splits as NESTOR. This is the standard protocol for evaluating existing methods on a new benchmark in machine learning.
I thank the authors for adding the suggested discussion of limitations. I acknowledge that incorporating sMRI information is an interesting direction. However, my main concern remains: the generalizability of the approach and the data beyond the very limited experimental setup of the dataset.
The authors state that their objective is predicting fMRI from cheaper modalities. In that case, they should demonstrate why the predicted fMRI is useful in the specific setup they are working on, as it has not been established that the approach generalizes beyond that setup.
What would make this compelling for me is a demonstration that this dataset can be leveraged to improve performance on other datasets with established benchmarks. This is especially important since the data was not collected but aggregated from existing sources, so showing that it is useful for other tasks would make the contribution meaningful.
In addition, comparing other methods on your dataset is problematic, as they were optimized for a different setting, and parameter tuning may improve them significantly.
Given the above, I maintain my score.
Response to Reviewer MKqT (1/2)
We appreciate the reviewer for the detailed feedback. Below are our responses to the specific points raised.
Re: The benchmark relies on a strong assumption that brain activity is the same across two different sessions of viewing the same stimulus. Furthermore, it has been demonstrated that responses differ between the first viewing of a stimulus and subsequent viewings. The paper acknowledges that session-specific spontaneous fluctuations cannot be predicted from EEG, but it assumes the stimulus-driven response is identical across sessions and does not address repeated-viewing effects or other variations due to internal state (e.g., attention, arousal, or fatigue). This should be disclosed in the paper as part of its limitations.
We agree and will discuss repeated-viewing effects and internal states in the Discussion (Section 6). We note that NEST requires only a shared stimulus-driven component rather than an identical response (Section 3.1).
Aligning separate-session recordings via a shared stimulus is common practice in multimodal neuroscience [R1-4], and prior studies show that the stimulus-driven response remains consistent across repeated viewings, although its strength decreases [R5-7]. Simultaneous EEG-fMRI avoids this assumption but introduces gradient and ballistocardiogram artifacts and requires specialized hardware, which limits cohort sizes [R8-9].
Our results show that NESTOR predicts the fMRI of unseen subjects from EEG recorded in a different session, and also on a film whose fMRI it never saw during training. This suggests the potential of using non-simultaneous recordings that follow standard clinical protocols and are much easier to collect at scale.
Reference
[R1] Cichy, Radoslaw Martin, et al. "Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence." Scientific reports 6.1 (2016): 27755.
[R2] Cichy, Radoslaw M., and Aude Oliva. "AM/EEG-fMRI fusion primer: resolving human brain responses in space and time." Neuron 107.5 (2020): 772-781.
[R3] Haufe, Stefan, et al. "Elucidating relations between fMRI, ECoG, and EEG through a common natural stimulus." NeuroImage 179 (2018): 79-91.
[R4] Hebart, Martin N., et al. "THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior." Elife 12 (2023): e82580.
[R5] Dmochowski, Jacek P., et al. "Correlated components of ongoing EEG point to emotionally laden attention–a possible marker of engagement?." Frontiers in human neuroscience 6 (2012): 112.
[R6] Golland, Yulia, et al. "Extrinsic and intrinsic systems in the posterior cortex of the human brain revealed during natural sensory stimulation." Cerebral cortex 17.4 (2007): 766-777.
[R7] Nastase, Samuel A., et al. "Measuring shared responses across subjects using intersubject correlation." Social cognitive and affective neuroscience 14.6 (2019): 667-685.
[R8] Ebrahimzadeh, Elias, et al. "Simultaneous electroencephalography-functional magnetic resonance imaging for assessment of human brain function." Frontiers in Systems Neuroscience 16 (2022): 934266.
[R9] Telesford, Qawi K., et al. "An open-access dataset of naturalistic viewing using simultaneous EEG-fMRI." Scientific Data 10.1 (2023): 554.