- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes BISURE (Bidirectional Sufficient Representation), a framework to unify brain and visual stimuli representations. BISURE makes brain and image embeddings interchangeable. The same representation space can be use for decoding visual images from brain signals as well as encoding brain responses to a stimulus. It also enables translating one participant's brain response to another participant's brain response to the same stimulus and even going from one modality to another (EEG and MEG). BISURE is evaluated on image retrieval and reconstruction tasks from EEG, MEG, and fMRI data (THINGS data). Encoding brain data from visual stimuli as well as from other participants brain responses is evaluated with Pearson correlation between the actual and predicted brain responses and is compared to noise ceiling. An ablation study is done on the effect of the "tie" term in the training objective.
Critical strengths
The problem addressed in the paper is an important step to understand and model the shared principles between representations of different modalities (EEG, MEG, fMRI, and images) and to model how these shared representations transform into each of the modalities. Solving this problem (i.e., demonstrating its generalization to unseen samples), can for example address data-scarcity problems for some modalities by allowing generating brain responses of the scarcer modality from the abundant one. The authors have shown that their proposed model effectively builds this shared representation space and enables translating between modalities even without paired data.
Critical weaknesses
The main issue of the paper is its language and writing. It uses uncommon words and phrases for a scientific paper with an inaccurate narrative language to describe the methods and results. Many parameters and notations used are not defined anywhere or anywhere close to where they are mentioned. For the architectures used the paper refers the readers to the appendix, but the same narrative and vague language, makes it extremely difficult to understand the details and sometimes even the high-level picture of the methodology. This also makes it really hard to assess the paper as it is extremely difficult to follow the paper's text.
The second weakness is that some of the choices are not supported in the paper. This ranges from choices such as the way equation 7 is defined (as a difference between two cosine similarity terms) to more specific ones such as how the hyper parameter values were selected (number of epochs, the architectures used, etc.) especially given that the authors insist that they have not use any test data for hyper parameter selection while they mention they have no validation set.
Finally some recent baselines such as BrainHive (Zhang et al. 2026) or Liu et al 2026 (https://ojs.aaai.org/index.php/AAAI/article/view/38813) with higher performances than the proposed method are not discussed in the paper.
Questions for authors
-
(Future suggestion) Many of the notations mentioned in the paper are not defined, at least close enough to where they are used. I had to search the whole paper for the meaning of those notations. Examples include but are not limited to: a. m, s in equation 2 (they are only defined at line 101), b. L, d in equation 4, c. d in equation 6. Some of these I could not find at all which I ask about them below.
-
What is a measurement interface T_m? Is it the modality tokenizer? It isn't defined anywhere.
-
What is image map g_phi? Again it's not defined anywhere.
-
How is d defined in equation 6? Is it the same as delta_Z in equation 7?
-
In lines 210-211, you mentioned "fMRI retains its released image split", why is this choice different than what was decided for EEG and MEG? Doesn't fMRI data share any image concept with EEG and MEG datasets?
-
Finally, while I generally understand the proposed method, based on the paper, I could not what specific architectures are used for brain tokenizers, brain encoders, mappings of brain and visual embeddings to the latent space (Z) and how exactly reconstruction works. It would be nice if the authors provided a figure to show visualize these, but, for now, I encourage the authors to at least provide clear descriptions of the components used with common network names (convolution, transformer, etc.) instead of vague narratives.
2 comments
Question 1: Notation (m, s in Eq. 2; L, d in Eq. 4; d in Eq. 6)
| symbol | meaning |
|---|---|
| m (Eq. 2) | recording modality, m ∈ {EEG, MEG, fMRI} |
| s (Eq. 2) | participant; X_{m,s} is the response of participant s recorded with modality m |
| L, d (Eq. 4) | Z_brain ∈ R^{L×d}: L = 256 latent tokens, each of width d = 256 |
| d (Eq. 6) | now named L_tie; see Question 4 |
Question 2: What is the measurement interface T_m?
It is the modality tokenizer, and the revision calls it that. It turns a response into tokens for the shared encoder:
| modality | tokenizer T_m | tokens |
|---|---|---|
| EEG | 63×250 response cut into temporal patches (length 10, stride 5), plus position and time embeddings | 3,087 |
| MEG | depthwise temporal convolutions (kernels 7/15/31), then patches | 6,504 |
| fMRI | participant-specific linear layer over the voxels | 1 |
Question 3: What is the image map g_phi?
A frozen CLIP image feature is projected to eight tokens; these and 256 learned queries pass through two transformer self-attention blocks, and the query outputs are Z_stim, the same shape as Z_brain.
Question 4: Is d in Eq. 6 the same as Δ_Z in Eq. 7?
No:
| L_tie in Eq. 6 (formerly d) | Δ_Z in Eq. 7 | |
|---|---|---|
| role | training loss | evaluation metric, never trained on |
| definition | distance between Z_stim and Z_brain (stop-gradient), per latent token: InfoNCE for EEG/MEG, 1 − cos for fMRI | matched-pair minus shuffled-pair cosine |
Question 5: Why fMRI keeps its released split (lines 210–211)
We follow BrainFLORA's split, the tri-modal setting closest to ours. THINGS-fMRI keeps its released test set because it is the only part of that dataset in which each image is repeated (12 times), which gives reliable test responses and keeps our fMRI numbers comparable with prior work. The datasets do share concepts, as you note:
| training set → test set | shared test concepts | shared test images |
|---|---|---|
| fMRI → EEG | 87 / 200 | 77 / 200 |
| fMRI → MEG | 86 / 200 | 1,032 / 2,400 |
| EEG, MEG → fMRI | 86 / 100 | 0 / 100 |
Setting. BiSURE retrained with every overlapping concept removed from training (3 seeds).
| BiSURE | overlap removed | |
|---|---|---|
| EEG top-1 ↑ | .594 ± .009 | .584 ± .025 |
| MEG top-1 ↑ | .179 ± .020 | .176 ± .014 |
| fMRI top-1 ↑ | .822 ± .027 | .824 ± .023 |
| EEG→MEG translation ↑ | .84 ± .02 | .81 ± .02 |
Result. All changes are within seed noise, so the results do not depend on the overlap. The revision adds both tables (Appendix D.5).
Question 6: Architectures and how reconstruction works
| component | architecture |
|---|---|
| shared encoder E_θ | Perceiver: 256 learned latents, eight blocks of cross-attention to the tokens followed by self-attention (width 256, 8 heads) |
| visual readout R | EEG/MEG: mean pooling and a two-layer MLP; fMRI: mean pooling, four residual MLP blocks and per-token projectors |
| response head D | coordinate decoder shared across participants, plus a small participant-specific head |
Reconstruction reuses the frozen model: the visual readout of Z_brain is mapped into the image embedding of a frozen unCLIP-SDXL generator (a ridge map fitted on training data, then a frozen image-only bridge), combined with a sample from a diffusion prior, and rendered. No neural parameter is retrained (Figure 10).
Thank you again for the detailed and constructive review. Your comments have made the paper substantially clearer. We hope the revision resolves your concerns about presentation and soundness; if so, we would be grateful if you would consider raising your score.
Thank you for the detailed review and for recognizing what BiSURE contributes: it addresses an important step toward modeling the shared principles of EEG, MEG, fMRI and image representations, with a direct use in data-scarce modalities, and it effectively builds this shared representation space and translates between modalities even without paired data.
Weakness 1: Language, notation, and the description of the architecture
We agree that the presentation needed work, and we have uploaded a revised PDF (changes in blue):
Questions 1–4 and 6 below give the definitions and architectures.
Weakness 2 (a): Why Eq. 7 subtracts two cosines
A raw cosine between Z_stim and Z_brain is dominated by directions shared by all pairs, such as a modality's mean, so it can be high even when an image's own response agrees no better than another image's. Subtracting the shuffled-pair cosine removes this common part: Δ_Z measures how much more a stimulus's own response agrees than another stimulus's.
Weakness 2 (b): Hyperparameter selection without a validation set
None of the three datasets defines a validation split. Following the convention of this benchmark, we chose the settings (architecture, training length) on test performance, as the public pipelines we compare against do; we never select checkpoints on test, and every BiSURE number is read from the final checkpoint. Our future work holds out a validation split from the start.
Setting. To measure what a validation split changes, we held out 10% of each modality's training concepts, retrained BiSURE (3 seeds), and selected each run's checkpoint on validation retrieval.
Result. Every change is below .02 and within seed noise, and the comparisons in Table 2 hold under either rule. The revision adds this check (Appendix D.4).
Weakness 3: Brain-HIVE and Liu et al. (2026)
Their higher retrieval numbers come from a different exam: both learn the retrieval gallery. An image tower trained jointly with the brain encoder moves the test images to where the brain signal reaches; Brain-HIVE's learned gallery places the 200 test images almost orthogonally (mean cosine .012, vs .62 in BiSURE's fixed CLIP gallery). With the gallery fixed, the advantage disappears:
Learning the gallery adds .25 top-1 to Brain-HIVE, and Liu et al.'s own ablation shows the same (80.0 → 56.2 without the learned image adapter). A learned gallery is a legitimate way to raise retrieval accuracy, but the score then no longer isolates the brain representation; BiSURE keeps one fixed target because the same latent is also read out for encoding and translation.