acceptodds
Withdrawn from ICLR 2027

Learning Neurovascular Codes for EEG-to-fMRI Synthesis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In this paper, we introduce **Neurovascular Codes** as shared EEG representations for fMRI generation. We first train a separate EEG encoder to predict each fMRI signal, learning target-specific temporal and spectral features from the same EEG window. After training, we freeze the encoders and project their concatenated hidden features into a shared neurovascular code. Conditioned on this code, *stochastic interpolant* transforms Gaussian noise into a vector of fMRI signals, allowing different responses to be sampled for the same EEG window. We evaluate our approach on eight public simultaneous EEG-fMRI datasets, including four resting-state and four task-based datasets with 61,646 volume-level pairs. Under within-subject and held-out-subject evaluation, our model improves average Pearson correlation over the strongest encoder baselines by 19.1% and 19.8%, respectively. The corresponding improvements over the strongest generator baselines are 18.8% and 47.9%. Beyond reconstruction accuracy, we also evaluate the generated signals using subject-identification classifiers trained on real fMRI. Our method achieves the highest mean accuracy among the evaluated generators. We hope that our work opens up new opportunities for cross-modal brain modeling.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG-to-fMRI, neurovascular codes, stochastic interpolants

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HdLLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce NVCode, an EEG-to-fMRI synthesis model trained using a two-stage approach:

  1. Training ROI-specific EEG encoders (for 6 pre-selected brain ROIs, and one global signal), in order to learn latent EEG representations per ROI. This is done using supervised EEG-fMRI pairs recorded simultaneously.

  2. Jointly train: (i) an EEG-representation projector on the concatenated learned EEG hidden features from Stage-1, to generate a “NeuroVascular code”; and (ii) an fMRI generator from noise using stochastic interpolation, conditioned on the learned “NeuroVascular code”. This stage generates the final fMRI signal.

The authors claim to learn a good representation of EEG, that supports high-accuracy fMRI generation. The model’s performance is evaluated on eight datasets separately, under both inter-subject and intra-subject settings. However, only separate components of the method (their EEG encoders, and the conditional generator) are compared against previous methods. Their end-to-end fMRI prediction from EEG (which is the stated goal of this model) is never compared against any of the competing methods.

Critical strengths
  • The paper introduces “neurovascular codes” to condition fMRI generators.
  • Evaluation and comparison of subcomponents of the architecture.
Critical weaknesses
  1. Important details and critical references are missing:
  • The fact that the mean fMRI signals of only 6 brain ROIs (+ one global) are being recovered by the proposed method is not mentioned explicitly anywhere in the main paper. It is implicitly hidden in Figs. 3 and 5 (using the strange term “target”, instead of “brain-ROI”). Note that recovering only 6 (+1) ROIs is in sharp contrast to previous full-brain SotA EEG2fMRI methods (e.g., UnEBOLT – see [1] below), which simultaneously predict fMRI activations of all brain-ROIs (up to 256). These full-brain methods are neither mentioned in the current paper, nor compared against.

  • The current method trains a separate EEG Encoder model for each of the 6 ROIs (+global signal). While this is feasible for 7 targets, it is unclear how this design will scale up to full brain settings with 64, 128 or 256 brain ROIs. Moreover, the claimed contribution of the authors regarding their NVcode being an important shared EEG representation across the 6 ROIs seems somewhat overstated, given that the full-brain EEG-to-fMRI methods learn a shared representation across all brain-ROIs.

  • Important missing/unclear details: (i) The EEG window size is not specified anywhere in the main paper. (ii) The exact definition of an “fMRI target” is never defined in the main paper. In particular, it is never mentioned that each “target” is a regional BOLD signal corresponding to a selected brain ROI, and that “N targets” is only N = 7, consisting of 6 ROIs plus one global signal. (iii) The experimental setting is very unclear – it is not clear from the main paper whether the Mean Pearson Correlation R in Table-1 is computed across time or not, nor what it is averaged across -- across targets? across subjects? splits? datasets? random seeds? Such important details must appear in the main body of the paper and cannot be hidden somewhere in the appendix.

  1. Lack of empirical comparison to prior work: There is no comparison of the full EEG-to-fMRI prediction accuracy of the proposed model against previous EEG-to-fMRI methods. Only partial/individual components of the model are ablated/compared, but not the end-to-end EEG2fMRI prediction. Without such a comparison, it is impossible to assess the performance of the method or the importance of the proposed NVcode. For this paper to be published, such end-to-end results vs. state-of-the-art competitive models (on the same datasets and configurations) must be added.

[1] UnEBOLT: A Unified Model for EEG-to-BOLD Translation and Functional Connectivity Reconstruction Yamin Li, Ange Lou, Chang Li, Shiyu Wang, Haatef Pourmotabbed, Ziyuan Xu, Shengchao Zhang, Dario J. Englot, Soheil Kolouri, Daniel Moyer, Roza G. Bayrak, Catie Chang Proceedings of the 9th International Conference on Medical Imaging with Deep Learning, PMLR 315:2338-2351, 2026.

Questions for authors
  1. Please compare NVCode’s end-to-end fMRI prediction with UnEBOLT (Li et al., 2026), NeuroBOLT (Li et al., 2025), and other recent end-to-end EEG-to-fMRI methods, under the same dataset and training/evaluation configuration used by these prior methods.

  2. Could you clarify your claim about broader “cross-modal representation learning” capability? How is this demonstrated? Since the EEG representations are learned using fMRI prediction and then evaluated again on EEG-to-fMRI generation, do you have evidence that these representations transfer to a different task, dataset, or modality?

  3. Did you experiment with jointly training your model on all 8 datasets together? How do these results compare to the results of your individually-trained models per dataset, which you then average across all models?

  4. Did you evaluate zero-shot or transfer-learning generalization across datasets or tasks? If so, how well do the learned representations transfer without retraining or with limited fine-tuning? (such results are reported in previous EEG2fMRI papers).

Reviewer_J2taOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NVCode for synthesizing fMRI signals from EEG. The framework has two stages: first, a separate EEG encoder is trained to predict each fMRI target signal. These encoders are then frozen and their features are concatenated and projected into the neurovascular code. Conditioned on this code, a flow-matching model generates fMRI signals from Gaussian noise.

Critical strengths
  1. A valuable benchmark. The paper unifies the target definitions, data splits and evaluation metrics across eight public EEG–fMRI datasets. If released, this benchmark could be a valuable reference for the community.
  2. Comprehensive baseline comparisons. The paper evaluates 12 encoder baselines and 7 generative baselines, and the appendix reports means and standard deviations over three random seeds.
Critical weaknesses
  1. Limited novelty. The proposed "stochastic interpolant" is essentially standard conditional flow matching. The neurovascular code is simply a concatenation of encoder features followed by a two-layer MLP. Despite its name, the model incorporates no biological prior related to neurovascular coupling, such as the hemodynamic response function.
  2. The comparison is not fair. In Table 2, the projector ψ∗\psi^* is jointly trained with NVCode's velocity field and then frozen and reused by all baseline generators. The baselines therefore receive conditions that were adapted to a different model.
  3. The subject identification experiment is logically flawed. If the generated fMRI signals faithfully preserve the characteristics of real fMRI, a classifier trained on real data would generally be expected to achieve performance comparable to that on real test data. Instead, every method scores higher than real fMRI. A possible explanation is that subject-specific cues such as electrode impedance and artifacts might be trained into the generated fMRI.
  4. The term "fMRI synthesis" may overstate the scope of the work. According to Appendix A.1, "Each volume defines a seven-dimensional target vector containing six DiFuMo ROIs and one global signal." This suggests that the model generates only 7-d ROI-level signals per time point, not full spatially resolved fMRI volumes. The 3D brain renderings (e.g., Figure 3, Figure 6) may be misleading.
Questions for authors
  1. Could the authors report the parameter counts of NVCode's seven EEG encoders and each baseline encoder? To ensure a fair comparison, please report the results of training 7 separate target-specific encoders for strong baselines such as SPaRCNet and NeuroBOLT.
  2. Without any generator, what Avg. R, CRPS and FC-MAE do the 7 first-stage linear heads achieve?
  3. Could the authors explain why all methods achieve higher subject identification accuracy than real fMRI data? Results under TrainSynth→TestReal protocol would be more compelling.
  4. Could the authors clarify what Cuneus, H Gyrus, Precuneus, etc., refer to in Figure 3(b)?
Reviewer_6d82OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes NVCode for predicting seven regional/global BOLD signals from EEG. Seven target-specific EEG encoders are trained separately, frozen, and combined through a learned shared projector. A conditional velocity model then transforms Gaussian samples into joint BOLD response vectors. Evaluation covers eight simultaneous EEG-fMRI datasets, within-subject and held-out-subject settings, marginal probabilistic scores, predictive-mean functional connectivity, and subject identification using a classifier trained on real fMRI. The reported aggregate held-out-subject correlation is 0.423, compared with 0.353 for the strongest encoder control and 0.286 for the strongest generator control.

Critical strengths
  1. Broad evaluation with useful controls

The evaluation across eight simultaneous EEG-fMRI datasets is a useful contribution in its own right. Acquisition conditions and cognitive states vary, and the paper reports both within-subject prediction and prediction for held-out subjects. The appendix specifies training-only normalization and a temporal buffer before within-subject testing (pp. 17–22). I find the held-out-subject result particularly relevant because a model that works only after fitting a new person's BOLD recordings has a narrower use case. The aggregate R of 0.423 is an encouraging result, although participant-level uncertainty is still needed to judge how consistently it is obtained.

  1. Evidence that the full output module is useful

Table 4 is more informative than the headline baseline rankings. With output modules matched in capacity, the full system obtains R = 0.423, compared with 0.317 for deterministic joint regression and 0.405 for independent interpolants (p. 9; pp. 23–24). The accompanying FC-MAE results support the practical benefit of the complete output module. The subject-identification probe adds a separate test of whether generated responses retain useful subject information: its classifier is trained on real fMRI. That probe has a clear, limited interpretation and should be retained alongside the reconstruction scores.

Critical weaknesses
  1. The headline comparisons do not isolate the claimed sources of improvement

I cannot yet tell how much of the gain is due to the proposed regional specialization and how much is due to the training resources assigned to the full system. NVCode uses seven independent encoders, whereas the encoder controls use one. Matching their concatenated output to 1,400 dimensions matches an interface, not total encoder capacity. The shared-encoder ablation changes capacity and target supervision together, so it does not separate those explanations (pp. 19, 23, 28).

The generator comparison has a related asymmetry. The context projector is jointly trained with NVCode and then frozen for every competing generator (pp. 20, 30). A representation selected by one generator's loss need not be equally suitable for another generator. The Rectified Flow row also uses a two-stage procedure with one reflow operation and divides the budget between the stages; NVCode directly fits the basic linear-path velocity objective. The original method treats reflow as an additional operation [1]. These results compare the disclosed configurations, but do not isolate the proposed design. A direct one-stage control with the same field and optimization, baseline-specific projector training, and a control with comparable total encoder capacity would address the attribution question. This matters because it determines what methodological lesson the benchmark actually supports.

  1. The stochastic joint-response motivation is only partly validated

The results persuade me more about predicting conditional means than about recovering a joint distribution of BOLD responses. R and FC-MAE are calculated from predictive means, while CRPS is reported marginally (p. 21). Good performance on these measures can coexist with poor conditional dependence among sampled regions. In Table 2, NVCode's inter-subject CRPS is 0.344, compared with 0.339 for Score SDE and 0.343 for EDM (p. 7). The probabilistic result therefore deserves a more qualified interpretation than the mean-prediction gain.

The independent-interpolant comparison is the natural place to test the joint-sampling claim. A multivariate proper score and a calibration analysis on held-out subjects would show whether joint access to the current response vector improves sampled dependencies or uncertainty. This is directly relevant to the stated motivation for modeling ambiguous responses to the same EEG. I would also want participant-based confidence intervals for the aggregate scores. Three training seeds on fixed splits describe optimization variability; they do not describe variation across participants or establish that a small retained dataset contributes a stable generalization result.

Questions for authors
  1. Would the generator ranking change if each method jointly trained its own context projector? Please include a direct one-stage rectified-flow control using the same velocity field, projector optimization and total budget as NVCode. For the encoder comparison, what are the total parameter counts, and does regional specialization still help against a shared or ensemble control with comparable capacity?

  2. For the full and independent-interpolant models, what do a multivariate proper score and held-out calibration show? I would find this more decisive than additional mean-reconstruction metrics because it tests the reason for generating joint samples. Please attach participant-based uncertainty to the comparison.

  3. Table 5 retains only two NatView scans. Please give the retained participant counts and exact split assignments for each corpus, including how those two scans enter held-out-subject evaluation. Does the aggregate advantage remain when NatView is excluded and when uncertainty is estimated over participants rather than training seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.