acceptodds
Under review as a conference paper at ICLR 2027

BiSURE: Toward a Unified Neural Representation for Visual Encoding and Decoding

Abstract

If visual encoding and decoding truly meet in the same interchangeable latent representation \(Z\), then a neural response from one participant should be readable, through \(Z\), as the response of another. We build and test such a representation. **BiSURE** (**B**idirectional **SU**fficient **RE**presentation) maps images and EEG, MEG, and fMRI responses into a common latent, from which the stimulus can be retrieved and every participant's measured response predicted, while retaining private interfaces for each instrument and participant. A single consistency term ties the image-derived and brain-derived routes, making their latents interchangeable. A response from one participant, passed through \(Z\) to another participant's response head, predicts that participant's response at 87–96% of the level reached from the image itself, with no stimulus and no parameter fitted for the pair, and more accurately than a linear map fitted for that pair on paired data. The same latent carries EEG responses into MEG at 84%, despite no paired EEG–MEG recordings, and transmits more than the model's own decoded CLIP features. Without the latent tie, retrieval remains essentially unchanged, yet the two routes become unrelated and translation falls to at most 6%; the tie also improves encoding by 15–31%. The same model matches or exceeds the strongest specialist in retrieval on every instrument, predicts every participant's responses above an established encoding baseline, and reconstructs images from the frozen latent.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Neural representation learning, Brain decoding, Neural encoding, EEG/MEG/fMRI

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_unRwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes BISURE (Bidirectional Sufficient Representation), a framework to unify brain and visual stimuli representations. BISURE makes brain and image embeddings interchangeable. The same representation space can be use for decoding visual images from brain signals as well as encoding brain responses to a stimulus. It also enables translating one participant's brain response to another participant's brain response to the same stimulus and even going from one modality to another (EEG and MEG). BISURE is evaluated on image retrieval and reconstruction tasks from EEG, MEG, and fMRI data (THINGS data). Encoding brain data from visual stimuli as well as from other participants brain responses is evaluated with Pearson correlation between the actual and predicted brain responses and is compared to noise ceiling. An ablation study is done on the effect of the "tie" term in the training objective.

Critical strengths

The problem addressed in the paper is an important step to understand and model the shared principles between representations of different modalities (EEG, MEG, fMRI, and images) and to model how these shared representations transform into each of the modalities. Solving this problem (i.e., demonstrating its generalization to unseen samples), can for example address data-scarcity problems for some modalities by allowing generating brain responses of the scarcer modality from the abundant one. The authors have shown that their proposed model effectively builds this shared representation space and enables translating between modalities even without paired data.

Critical weaknesses

The main issue of the paper is its language and writing. It uses uncommon words and phrases for a scientific paper with an inaccurate narrative language to describe the methods and results. Many parameters and notations used are not defined anywhere or anywhere close to where they are mentioned. For the architectures used the paper refers the readers to the appendix, but the same narrative and vague language, makes it extremely difficult to understand the details and sometimes even the high-level picture of the methodology. This also makes it really hard to assess the paper as it is extremely difficult to follow the paper's text.

The second weakness is that some of the choices are not supported in the paper. This ranges from choices such as the way equation 7 is defined (as a difference between two cosine similarity terms) to more specific ones such as how the hyper parameter values were selected (number of epochs, the architectures used, etc.) especially given that the authors insist that they have not use any test data for hyper parameter selection while they mention they have no validation set.

Finally some recent baselines such as BrainHive (Zhang et al. 2026) or Liu et al 2026 (https://ojs.aaai.org/index.php/AAAI/article/view/38813) with higher performances than the proposed method are not discussed in the paper.

Questions for authors
  1. (Future suggestion) Many of the notations mentioned in the paper are not defined, at least close enough to where they are used. I had to search the whole paper for the meaning of those notations. Examples include but are not limited to: a. m, s in equation 2 (they are only defined at line 101), b. L, d in equation 4, c. d in equation 6. Some of these I could not find at all which I ask about them below.

  2. What is a measurement interface T_m? Is it the modality tokenizer? It isn't defined anywhere.

  3. What is image map g_phi? Again it's not defined anywhere.

  4. How is d defined in equation 6? Is it the same as delta_Z in equation 7?

  5. In lines 210-211, you mentioned "fMRI retains its released image split", why is this choice different than what was decided for EEG and MEG? Doesn't fMRI data share any image concept with EEG and MEG datasets?

  6. Finally, while I generally understand the proposed method, based on the paper, I could not what specific architectures are used for brain tokenizers, brain encoders, mappings of brain and visual embeddings to the latent space (Z) and how exactly reconstruction works. It would be nice if the authors provided a figure to show visualize these, but, for now, I encourage the authors to at least provide clear descriptions of the components used with common network names (convolution, transformer, etc.) instead of vague narratives.

2 comments

Thank you for the detailed review and for recognizing what BiSURE contributes: it addresses an important step toward modeling the shared principles of EEG, MEG, fMRI and image representations, with a direct use in data-scarce modalities, and it effectively builds this shared representation space and translates between modalities even without paired data.

Weakness 1: Language, notation, and the description of the architecture

We agree that the presentation needed work, and we have uploaded a revised PDF (changes in blue):

  • An architecture figure in the main text (Figure 2), mapping each component's name to its symbol, and detailed figures of every component (Figures 8–9) and of the reconstruction pipeline (Figure 10).
  • Every symbol is defined where it first appears, and every abbreviation is written out at its first use.
  • Eq. 6 now names its terms L_align and L_tie, and the "measurement interface" is called a tokenizer.

Questions 1–4 and 6 below give the definitions and architectures.

Weakness 2 (a): Why Eq. 7 subtracts two cosines

A raw cosine between Z_stim and Z_brain is dominated by directions shared by all pairs, such as a modality's mean, so it can be high even when an image's own response agrees no better than another image's. Subtracting the shuffled-pair cosine removes this common part: Δ_Z measures how much more a stimulus's own response agrees than another stimulus's.

Weakness 2 (b): Hyperparameter selection without a validation set

None of the three datasets defines a validation split. Following the convention of this benchmark, we chose the settings (architecture, training length) on test performance, as the public pipelines we compare against do; we never select checkpoints on test, and every BiSURE number is read from the final checkpoint. Our future work holds out a validation split from the start.

Setting. To measure what a validation split changes, we held out 10% of each modality's training concepts, retrained BiSURE (3 seeds), and selected each run's checkpoint on validation retrieval.

test top-1 ↑, mean ± sdEEGMEGfMRI
final checkpoint.576 ± .031.173 ± .017.821 ± .014
validation-selected checkpoint.591 ± .034.177 ± .025.802 ± .007

Result. Every change is below .02 and within seed noise, and the comparisons in Table 2 hold under either rule. The revision adds this check (Appendix D.4).

Weakness 3: Brain-HIVE and Liu et al. (2026)

Their higher retrieval numbers come from a different exam: both learn the retrieval gallery. An image tower trained jointly with the brain encoder moves the test images to where the brain signal reaches; Brain-HIVE's learned gallery places the 200 test images almost orthogonally (mean cosine .012, vs .62 in BiSURE's fixed CLIP gallery). With the gallery fixed, the advantage disappears:

EEG retrieval, fixed gallerytop-1 ↑
Brain-HIVE, our re-run of the released code (10 participants × 3 seeds).439
BiSURE.594 ± .009

Learning the gallery adds .25 top-1 to Brain-HIVE, and Liu et al.'s own ablation shows the same (80.0 → 56.2 without the learned image adapter). A learned gallery is a legitimate way to raise retrieval accuracy, but the score then no longer isolates the brain representation; BiSURE keeps one fixed target because the same latent is also read out for encoding and translation.

Question 1: Notation (m, s in Eq. 2; L, d in Eq. 4; d in Eq. 6)

symbolmeaning
m (Eq. 2)recording modality, m ∈ {EEG, MEG, fMRI}
s (Eq. 2)participant; X_{m,s} is the response of participant s recorded with modality m
L, d (Eq. 4)Z_brain ∈ R^{L×d}: L = 256 latent tokens, each of width d = 256
d (Eq. 6)now named L_tie; see Question 4

Question 2: What is the measurement interface T_m?

It is the modality tokenizer, and the revision calls it that. It turns a response into tokens for the shared encoder:

modalitytokenizer T_mtokens
EEG63×250 response cut into temporal patches (length 10, stride 5), plus position and time embeddings3,087
MEGdepthwise temporal convolutions (kernels 7/15/31), then patches6,504
fMRIparticipant-specific linear layer over the voxels1

Question 3: What is the image map g_phi?

A frozen CLIP image feature is projected to eight tokens; these and 256 learned queries pass through two transformer self-attention blocks, and the query outputs are Z_stim, the same shape as Z_brain.

Question 4: Is d in Eq. 6 the same as Δ_Z in Eq. 7?

No:

L_tie in Eq. 6 (formerly d)Δ_Z in Eq. 7
roletraining lossevaluation metric, never trained on
definitiondistance between Z_stim and Z_brain (stop-gradient), per latent token: InfoNCE for EEG/MEG, 1 − cos for fMRImatched-pair minus shuffled-pair cosine

Question 5: Why fMRI keeps its released split (lines 210–211)

We follow BrainFLORA's split, the tri-modal setting closest to ours. THINGS-fMRI keeps its released test set because it is the only part of that dataset in which each image is repeated (12 times), which gives reliable test responses and keeps our fMRI numbers comparable with prior work. The datasets do share concepts, as you note:

training set → test setshared test conceptsshared test images
fMRI → EEG87 / 20077 / 200
fMRI → MEG86 / 2001,032 / 2,400
EEG, MEG → fMRI86 / 1000 / 100

Setting. BiSURE retrained with every overlapping concept removed from training (3 seeds).

BiSUREoverlap removed
EEG top-1 ↑.594 ± .009.584 ± .025
MEG top-1 ↑.179 ± .020.176 ± .014
fMRI top-1 ↑.822 ± .027.824 ± .023
EEG→MEG translation ↑.84 ± .02.81 ± .02

Result. All changes are within seed noise, so the results do not depend on the overlap. The revision adds both tables (Appendix D.5).

Question 6: Architectures and how reconstruction works

componentarchitecture
shared encoder E_θPerceiver: 256 learned latents, eight blocks of cross-attention to the tokens followed by self-attention (width 256, 8 heads)
visual readout REEG/MEG: mean pooling and a two-layer MLP; fMRI: mean pooling, four residual MLP blocks and per-token projectors
response head Dcoordinate decoder shared across participants, plus a small participant-specific head

Reconstruction reuses the frozen model: the visual readout of Z_brain is mapped into the image embedding of a frozen unCLIP-SDXL generator (a ridge map fitted on training data, then a frozen image-only bridge), combined with a sample from a diffusion prior, and rendered. No neural parameter is retrained (Figure 10).

Thank you again for the detailed and constructive review. Your comments have made the paper substantially clearer. We hope the revision resolves your concerns about presentation and soundness; if so, we would be grateful if you would consider raising your score.

Reviewer_F1reOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BiSURE, a method for learning a shared representation across EEG, MEG, and fMRI for visual encoding and decoding. The method combines modality-specific interfaces, a shared neural encoder, an image-to-latent mapping, and a consistency objective that aligns image-derived and brain-derived representations. Experiments on THINGS datasets evaluate image retrieval, neural response prediction, cross-participant response translation, and image reconstruction. Results show competitive performance against modality-specific models and demonstrate that the consistency objective enables response translation across participants and between EEG and MEG.

Critical strengths
  • The paper evaluates shared representations through cross-participant response prediction and route-swapping experiments. The consistency-objective ablation and CLIP-route comparisons provide useful evidence that the learned latent supports response translation.
Critical weaknesses
  1. The benefit of combining three neural modalities is insufficiently established. Prior work [1, 2] has already explored shared representations across EEG, MEG, and fMRI. Section 5.4 reports joint-training encoding gains of 0.005, 0.042, and 0.008 for EEG, MEG, and fMRI, respectively, without consistent retrieval improvements. The larger MEG gain is interesting, but the paper does not establish what complementary information the modalities contribute or whether all three are necessary.
  2. The interpretation of the shared latent remains insufficiently supported. The authors distinguish CLIP supervision from the learned neural latent, but the representation is still trained against pretrained visual features with their own inductive biases. Although the CLIP-route baseline is weaker, this does not establish that the latent captures modality-independent neural information rather than a richer stimulus-aligned representation. I do not see sufficient evidence supporting the stronger interpretation.
  3. The cross-modality evidence is incomplete. Cross-instrument response translation is demonstrated only between EEG and MEG, with markedly asymmetric performance (84% versus 43%). Translation involving fMRI is absent, and all target participants have response heads trained on their own data. The results establish transfer among trained participants but provide limited support for a unified representation across all three modalities. [1] Ferrante, M., Boccato, T., Rashkov, G., & Toschi, N. (2025). Towards neural foundation models for vision: Aligning EEG, MEG, and fMRI representations for decoding, encoding, and modality conversion. Information Fusion, 103650. [2] Li, D., Qin, H., Wu, M., Wei, C., & Liu, Q. (2025, October). Brainflora: Uncovering brain concept representation via multimodal neural embeddings. In Proceedings of the 33rd ACM International Conference on Multimedia (pp. 5577-5586).
Questions for authors
  1. Section 5.4 shows limited joint-training gains outside MEG. What complementary information does each modality contribute, and do pairwise-modality ablations support the necessity of all three?
  2. The latent is trained under pretrained visual supervision. What evidence distinguishes shared neural information from stimulus-aligned representations shaped by the visual model?
  3. Cross-instrument translation is demonstrated only between EEG and MEG. What evidence supports extending the interchangeability and bidirectional sufficiency claims to all three modalities?
4 comments

Thank you for the review and for recognizing BiSURE's two central pieces of evidence: it evaluates shared representations through cross-participant response prediction and route swapping, and the consistency-objective ablation and CLIP-route comparisons show that the learned latent supports response translation.

Your three questions led to new experiments. In short: EEG lifts MEG in a model trained on the two alone; translation comes from the tie, not from shared images or from visual alignment alone; and translation extends to new participants and into fMRI, with no paired recordings.

Conventions. All new models keep BiSURE's configuration and training budget and are read at their final checkpoint. Values are mean ± sd over 3 seeds; the few arms marked with 2 seeds are those for which the rebuttal period allowed no more runs. Translation encodes one participant's response and reads it out through another participant's response head, with nothing fitted for the pair; it is reported as the fraction of the target's own image-driven prediction that is reached (1.0 = as accurate as predicting from the image).

Weakness 1 and Question 1: What combining modalities contributes; pairwise ablation

Setting. A model trained on EEG + MEG only (no fMRI), with the same number of training steps per modality as BiSURE, compared with single-modality models and with BiSURE.

trained onMEG encoding r ↑fMRI encoding r ↑EEG→MEG translation ↑
that modality alone.083 ± .003.195 ± .005—
EEG + MEG.129 ± .003—.83 ± .02
EEG + MEG + fMRI (BiSURE).125 ± .004.203 ± .003.84 ± .02

Result.

  • EEG lifts MEG: the data-scarcest modality gains about 50% in encoding from joint training, with or without fMRI. Abundant, inexpensive recordings (EEG) thus improve the model of a scarce, expensive one (MEG).
  • fMRI completes the space at no cost: adding it keeps MEG encoding and EEG→MEG translation unchanged, raises fMRI's own encoding, and gives one latent that decodes and predicts responses in all three instruments.

The revision states Section 5.4 and the introduction in these terms.

Prior work [1, 2]. Both map each modality separately onto CLIP's image space with a contrastive loss, so their shared space is CLIP's rather than a learned representation of the neural responses, and neither predicts measured responses from it ([1]'s encoding and modality conversion retrieve the nearest stored neural embedding; [2] adds task decoders). BiSURE instead learns its own brain-derived latent, uses CLIP only as the retrieval target read out from it, and ties an image route and a response head to that latent, which is what makes route swapping and pair-free translation possible.

Weakness 2 and Question 2: Neural information vs a stimulus-aligned representation

Z is trained toward a visual target by design; the question is what the tie adds beyond visual alignment. Two tests answer it.

Test 1, a matched CLIP bottleneck. The no-tie model N has the same architecture, data, CLIP targets and response heads, trained without the tie. Its CLIP route decodes the source response to CLIP and predicts the target's response from it. We compare the correlation of each translated response with the target's measured response.

translation r ↑BiSURE latentN, CLIP bottleneck
EEG→EEG.0906 ± .0003.0767 ± .0015
MEG→MEG.1088 ± .0026.0818 ± .0025
fMRI→fMRI.1946 ± .0022.1565 ± .0024
EEG→MEG.1059 ± .0031.0857 ± .0012
MEG→EEG.0427 ± .0019.0420 ± .0007

Test 2, no shared training images. EEG and MEG retrained on disjoint images (no image seen by both) or disjoint concepts, against a control with the same halved data and shared images.

EEG→MEG translation ↑shared images (control)disjoint imagesdisjoint conceptsno tie
mean ± sd.78 ± .05 (2 seeds).70 ± .07.70 ± .03 (2 seeds)≤ .06

Result. BiSURE's latent translates better than the matched CLIP bottleneck in four of five directions (level on MEG→EEG), and translation survives without any shared image or concept, while it vanishes without the tie. We therefore state the claim as: a stimulus-aligned latent that the tie makes directly translatable between participants and between EEG and MEG, matching or exceeding decoding to CLIP.

Weakness 3 and Question 3 (a): Translation involving fMRI

BiSURE unifies what is represented, not how it is measured (Eq. 3): the latent keeps a modality-specific part that each instrument's head reads. Separating the two is what makes the representation useful: what an image evokes is shared across people and instruments, while how it is measured differs, so one latent can pool every instrument's data (letting abundant recordings inform scarce ones) and carry a response from any participant or instrument to another. EEG and MEG latents are close enough to be read directly (EEG→MEG .84 ± .02). The fMRI latent carries a much larger modality-specific part (about 17 times the norm), which an EEG or MEG latent does not have.

Setting. Into fMRI, a linear regression fitted only on the source participant's own training data maps the source latent onto the image route, removing the modality-specific part (no fMRI data, no paired recordings). Scored on the 14 concepts shared by the EEG/MEG and fMRI test sets.

translation into fMRI ↑BiSURE latent (no paired data)reference: decode to CLIP, then image route
MEG→fMRI.38 ± .05.35 ± .06
EEG→fMRI.21 ± .16.29 ± .06

Result. Translation into fMRI works without paired data, on par with decoding to CLIP; out of fMRI, the latent is read directly (fMRI→EEG .19 ± .08, fMRI→MEG .40 ± .23).

Weakness 3 (b): All target participants were seen in training

Setting. BiSURE retrained three times, each time holding out one EEG, one MEG and one fMRI participant. For a new target we fit only their own response head and participant embedding on their own data; the shared encoder, image map and decoder are not retrained. This per-person part is unavoidable for predicting a specific person's measured response: every brain is different (h_{m,s} in Eq. 3), so every encoding model fits it per person.

translation ↑, mean ± sd over 3 foldsnew participanttrained participants
into a new participant: EEG→EEG.82 ± .06.90 ± .02
into a new participant: MEG→MEG.61 ± .08.71 ± .11
into a new participant: EEG→MEG.69 ± .09.76 ± .10
from a new participant (nothing fitted): EEG→EEG.56 ± .03.92 ± .02

With about 11 minutes of a new EEG participant's data used to fine-tune the brain side, translation from them rises from .56 ± .03 to .76 ± .04, and with all of their data to .90 ± .02 (participant trained in: .92 ± .02).

Weakness 3 (c): The 84% vs 43% asymmetry

The asymmetry comes from the source, not from the latent. An MEG test response averages 12 different images of a concept, while the EEG target is one image: even the images themselves, averaged per concept, reach only .80 when read by the EEG heads. MEG is also the data-scarcest modality, with the smaller stimulus-specific part in its latent. With EEG as the source, translation reaches .84.

Thank you again for the thoughtful questions. They led to the experiments that now pin down what joint training and the tie contribute. We hope these results address your concerns, and we would be grateful if you would consider raising your score.

Thanks for the useful reply. Most of my previous concerns have been resolved, though the interpretation of the shared latent remains insufficiently supported. In addition, the results need proper explanation or analysis rather than simply being presented. I will maintain my original rating.

Thank you for the helpful feedback. Below, we address the interpretation of the shared latent with a new analysis and explain our results further.

1. What the shared latent represents

The latent is trained against visual features, so it is stimulus-aligned by construction. The question is whether it is only the visual model's representation of the stimulus, or also carries what brains share beyond it. We claim the latter and test it directly.

Analysis. Let T be the response translated from another participant, B the target participant's measured response, and C a prediction to remove. For each unit u (a sensor at one time point, or a voxel), across test stimuli, the correlation of T with B splits exactly into a part carried by C and the rest:

r(T,B)=r(T,C),r(C,B)⏟carried by C+[r(T,B)−r(T,C),r(C,B)]⏟beyond Cr(T,B)=\underbrace{r(T,C)\\,r(C,B)} _{\text{carried by }C}+\underbrace{\left[r(T,B)-r(T,C)\\,r(C,B)\right]} _{\text{beyond }C}

The second term is the covariance of the residuals of T and B after each is regressed on C, in units of their original sds, so it is a part of r(T, B). We report its share, averaged over units and participant pairs:

share beyond C;=;meanu[,ru(T,B)−ru(T,C),ru(C,B),]meanu,ru(T,B)\text{share beyond } C\\;=\\;\frac{\text{mean} _{u}\big[\\,r _{u}(T,B)-r _{u}(T,C)\\,r _{u}(C,B)\\,\big]}{\text{mean} _{u}\\,r _{u}(T,B)}

for two choices of C:

  • image-predicted: the target's response predicted from the image (image → CLIP → image map → target's head);
  • CLIP-decoded: the source's response decoded to CLIP, then mapped the same way, i.e. what passes between brains through CLIP's space.

A shuffled source gives the chance level. Source and target are different people, so their measurement noise is independent and does not contribute to the second term on average.

share beyond C ↑, mean ± sd over 3 seedsC = image-predictedC = CLIP-decodedshuffled source
EEG→EEG28 ± 0%24 ± 3%1%
MEG→MEG55 ± 3%23 ± 4%−2%
fMRI→fMRI34 ± 2%32 ± 1%0%
EEG→MEG49 ± 2%33 ± 4%0%
MEG→EEG23 ± 1%20 ± 5%1%

What this shows. Whichever prediction is removed, 20–55% of the translation correlation remains, positive in every participant pair at every seed (except 3 of 40 MEG→EEG pairs), while the shuffled source stays near zero. The CLIP-decoded control, which also uses the source's response, removes more in every direction. We believe this remainder reflects responses that brains share but the CLIP-based predictions do not capture: it is shared between different people, so it is unlikely to be noise, and between an EEG and an MEG participant, so it does not appear specific to one instrument. It is still evoked by the stimulus; our claim is that the latent carries what brains share about the stimulus beyond what is predicted from CLIP features.

2. Explanation of the results in our response

  • Combining modalities. MEG's data are the scarcest. Its tokenizer is its own, but the Perceiver encoder and the image map are shared, so they are also trained on EEG's far larger data. With joint training, MEG's brain and image latents agree 1.5× more (ΔZ\Delta_Z .14 → .21), consistent with its 50% encoding gain.
  • Neural information versus visual alignment. The CLIP route passes only what the source's response says about CLIP features, while the latent also passes the shared content measured above, consistent with its higher translation than the CLIP bottleneck (4 of 5 directions). The tie pulls the image-derived latent toward the brain-derived one, which the stop-gradient keeps as the reference. The response heads are trained only on the image route, so it predicts responses on its own, and the heads learn the brain latent's coordinates. These are common to all participants, because one Perceiver encoder maps every response toward the same visual target. So translation needs no shared training images (EEG→MEG .70). Without the tie, the heads' coordinates differ from the brain latent's, so translation is near zero although retrieval is unchanged. Figure 4 shows both.
  • Translation involving fMRI. Translation into fMRI needs no paired data: on 87 held-out concepts, EEG and MEG responses translated into fMRI through the CLIP route reach 44–65% of the image-driven prediction, level with or above a regression fitted on paired images. The raw latent does not transfer because the fMRI latent's modality-specific part is much larger; projecting it onto the source's image route first fixes this.
  • New participants. Every brain is different. Into a new participant, fitting only their own response head keeps translation near the trained level. From a new EEG participant, with nothing fitted, translation reaches 56% of the image-driven prediction (trained participant: 92%), likely as the encoder never saw them; fine-tuning only the brain side on 11 minutes of their data raises it to 76%.

Most of these results are in the revised paper (Sections 5.2 and 5.4; new material in blue: Figure 4 and Appendices F.1 and G).

Reviewer_7gYDOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents a joint training approach for mapping brain activity (EEG, MEG and fMRI) and natural images to the same space, enabling both brain decoding and encoding across device types and subjects. The approach learns a shared representation thanks to the addition of a consistency term in the global loss function. Experiments on three datasets with the same stimulation images (i.e. from the THINGS database) show that the shared representation can be used to predict a subject’s brain activity from another’s, yielding a high fraction of the noise ceiling; similarly, EEG can be used to predict an MEG response. Final model performance is competitive with or outperforms existing baselines on image retrieval, image reconstruction, and EEG/MEG/fMRI encoding, under a common evaluation scheme.

Critical strengths
  1. The paper presents a model with strong performance on three different brain modalities, in both decoding and encoding directions, outperforming existing methods in most comparisons. Along with a strongly motivated approach and loss function, and multiple relevant ablation studies, this is overall a convincing account of a method for unifying neural representations for decoding and encoding.
Critical weaknesses

None

Questions for authors
  1. Why is the stop-gradient term on ZbrainZ_{brain} and not on ZstimZ_{stim}? Would results be negatively affected by the alternative?

  2. Suggestion for future work: it would be interesting to see 2D projections of the learned feature space, color-coded by the different dimensions/variables at play (brain device, input type, subject, etc.).

3 comments

Thank you for the positive assessment and for recognizing BiSURE's strong performance on three brain modalities in both the decoding and the encoding direction, its strongly motivated approach and loss function, and its relevant ablation studies. Both of your questions led to new results.

Question 1: Why is the stop-gradient on ZbrainZ_{brain} and not on ZstimZ_{stim}? Would the alternative hurt results?

BiSURE sets out to learn a representation of brain responses, so the brain-derived latent ZbrainZ_{brain} is the reference and the image route ZstimZ_{stim} is trained to follow it. A stop-gradient on ZstimZ_{stim} would instead let the image model shape the latent, which is not the representation BiSURE aims to learn; for this reason we do not study that variant. We tested instead whether the stop-gradient is needed at all.

Setting. BiSURE retrained without the stop-gradient (gradients through both routes), with the same configuration and training budget; 3 seeds each.

mean ± sd, 3 seedswith stop-gradient (BiSURE)without stop-gradient
fMRI ΔZΔ_Z ↑.52 ± .03.09 ± .02
fMRI encoding r ↑.203 ± .003.174 ± .003
EEG→MEG translation ↑.84 ± .02.79 ± .03
retrieval top-1 ↑, EEG.594 ± .009.597 ± .008
retrieval top-1 ↑, MEG.179 ± .020.177 ± .014
retrieval top-1 ↑, fMRI.822 ± .027.804 ± .028

ΔZΔ_Z (Eq. 7) measures how much more the image-derived and brain-derived latents of the same stimulus agree than those of different stimuli; translation is the fraction of the target's image-driven prediction that is reached.

Result. Without the stop-gradient, the fMRI latent loses its stimulus-specific part (ΔZΔ_Z .52 → .09), fMRI encoding drops, and EEG→MEG translation drops; retrieval top-1 is unchanged within seed noise. The stop-gradient on ZbrainZ_{brain} is what keeps the shared latent stimulus-specific.

Question 2 (suggestion): 2D projections of the learned feature space

Thank you for the suggestion. We added it to the main text as Figure 4 (UMAP of the EEG test set):

  • (a, b) Image route vs brain route, with and without the tie. With the tie, the two latents of each image coincide; without it, the two routes form separate clusters.
  • (c, d) Ten participants, raw EEG vs BiSURE's latent. Responses of different participants to the same image cluster together in the latent but not in raw EEG.

Thank you again for the supportive review. Your questions added a new ablation and a new main-text figure, and both confirm the paper's design. If you feel these additions strengthen the paper further, we would be grateful if you would reflect this in your final comment.

For your reference, we also trained the variant you asked about, with the stop-gradient on ZstimZ_{\mathrm{stim}} instead of ZbrainZ_{\mathrm{brain}} (same configuration and training budget; one seed, as the remaining rebuttal time allowed), in the hope that it answers your question more fully.

stop-gradient on ZbrainZ_{\mathrm{brain}} (BiSURE, 3 seeds)stop-gradient on ZstimZ_{\mathrm{stim}} (1 seed)
encoding r ↑, EEG.099 ± .001.085
encoding r ↑, MEG.125 ± .004.107
encoding r ↑, fMRI.203 ± .003.151
EEG→MEG translation ↑.84 ± .02.58
ΔZ\Delta_Z ↑, EEG / MEG / fMRI.34 / .21 / .52.15 / .09 / .46
retrieval top-1 ↑, EEG.594 ± .009.556
retrieval top-1 ↑, MEG.179 ± .020.189
retrieval top-1 ↑, fMRI.822 ± .027.817

Result. Yes, it would hurt. With the stop-gradient on ZstimZ_{\mathrm{stim}}, the image map no longer learns from the tie: encoding drops in all three modalities, the two latents agree less (lower ΔZ\Delta_Z everywhere), and EEG→MEG translation falls from .84 to .58. Retrieval stays close, except EEG top-1 (−.04). Placing the stop-gradient on ZbrainZ_{\mathrm{brain}} is therefore better on every axis the paper uses, in line with the design rationale in our response.

Thank you again for the thoughtful question.

Thank you to the authors for the additional clarifications and results on the stop-gradient and on UMAP projections. I maintain my original rating.

Reviewer_MnnCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BiSURE, a unified framework for visual neural encoding and decoding across EEG, MEG, and fMRI. The central objective is to learn a shared latent representation that supports both predicting neural responses from visual stimuli and recovering stimulus information from the measured neural activity.

The framework combines a shared neural encoder, a CLIP-based image-to-latent mapping, a visual retrieval head, and modality- and participant-aware neural response prediction components. Its key mechanism is a latent consistency objective that explicitly aligns the stimulus-derived and brain-derived representations. Importantly, the retrieval head is trained only on brain-derived latents, enabling route swapping to serve as a test of representational interchangeability.

The authors evaluate BiSURE on THINGS-based EEG, MEG, and fMRI datasets. The experiments include stimulus retrieval, neural encoding, latent agreement, route swapping, cross-participant and cross-modality response translation, and image reconstruction. The empirical results show that removing the latent consistency term largely preserves retrieval performance but substantially degrades route swapping and neural response translation. With the loss, within-modality cross-participant translation achieves approximately 87% to 96% of the corresponding image-driven encoding performance without pair-specific fitting, while EEG-to-MEG translation reaches approximately 84%. The framework also achieves competitive retrieval performance and improves neural encoding correlations over the evaluated baselines.

Overall, the paper investigates whether a shared latent representation can support both encoding and decoding and uses route swapping and neural response translation to test this property.

Critical strengths

1. A convincing evaluation of representational interchangeability: The strongest contribution is that the paper goes beyond evaluating encoding and decoding performance separately and directly tests whether stimulus-derived and brain-derived latents can be used interchangeably. The latent-tie ablation is particularly convincing. These results directly support the paper’s central claim, while the use of disjoint trial halves strengthens the experimental validity.

2. Strong empirical validation across modalities and participants: The paper provides extensive experiments across EEG, MEG, and fMRI, achieving competitive retrieval performance and improved neural encoding across all 17 participants. Particularly notable are the cross-participant and cross-modality translation results, which require no pair-specific fitting. Comparisons with pair-fitted ridge regression and a decoded-CLIP route further support the value of the learned latent beyond conventional retrieval representations.

Critical weaknesses

1. Limited generality of cross-participant and cross-modal translation: All evaluated participants are included during training, so generalization to unseen individuals remains untested. Cross-modal translation is limited to EEG and MEG, with substantial directional asymmetry (84% vs. 43%), and no translation involving fMRI is reported. Moreover, EEG and MEG share 16,520 training images, making it difficult to isolate the contribution of shared visual supervision. These limitations weaken the broader claim of a unified representation across participants and modalities.

2. Insufficient matched baselines to establish the contribution of the learned latent representation: The no-tie ablation demonstrates the importance of latent coupling within BiSURE but does not establish its advantage over simpler shared-representation approaches. The pair-fitted regression baseline lacks comparable visual supervision and model capacity, while the CLIP-route comparison relies on BiSURE's own decoded features and additional mappings. A capacity- and supervision-matched CLIP-bottleneck baseline would better establish whether the improvements arise from the proposed latent representation rather than conventional stimulus-aligned representations.

Questions for authors

Q1: Since fMRI retains a separate image split, are any test images or concepts from EEG/MEG included in fMRI training, or vice versa? If so, could the authors quantify this overlap and clarify its potential effect on the joint model's evaluation?

Q2: Could the authors compare BiSURE with an independently trained, capacity- and supervision-matched CLIP-based shared representation that supports both neural decoding and response prediction? This would help isolate the benefit of BiSURE's proposed latent beyond conventional visual alignment.

Q3: Why is cross-modality translation evaluated only between EEG and MEG, despite the claim of a unified representation across EEG, MEG, and fMRI? Can the framework's interchangeability be evaluated for translations involving fMRI?

Q4: Appendix C.1 states that the temperature is optimized on the complete gallery. Does this optimization use test-set labels? If so, could the authors report information gain using temperatures calibrated on independent data?

3 comments

Thank you for the careful review and for recognizing the two strengths at the core of BiSURE:

  • A direct test of representational interchangeability: route swapping and response translation, with the latent-tie ablation, directly support the central claim.
  • Validation across modalities and participants: competitive retrieval on EEG, MEG and fMRI, improved encoding for all 17 participants, and translation with no pair-specific fitting (87–96% across participants, 84% EEG→MEG).

Each of your questions became a new experiment, and together they show that translation through BiSURE's latent survives every control you proposed: it extends to new participants, needs no shared training images, does not depend on the cross-dataset overlap, and matches or exceeds a matched CLIP bottleneck.

Conventions. All new models keep BiSURE's configuration and training budget and are read at their final checkpoint. Values are mean ± sd over 3 seeds; the few arms marked with 2 seeds are those for which the rebuttal period allowed no more runs. Translation encodes one participant's response and reads it out through another participant's response head, with nothing fitted for the pair; it is reported as the fraction of the target's own image-driven prediction that is reached (1.0 = as accurate as predicting from the image).

Weakness 1 (a): Generalization to unseen participants

Setting. We retrained BiSURE three times, each time holding out one EEG, one MEG and one fMRI participant. For a new target participant we fit only their own response head and participant embedding on their own data; the shared encoder, image map and decoder are not retrained. This per-person part is unavoidable for predicting a specific person's measured response: every brain is different (h_{m,s} in Eq. 3), so every encoding model fits it per person.

translation ↑, mean ± sd over 3 foldsnew participanttrained participants
into a new participant: EEG→EEG.82 ± .06.90 ± .02
into a new participant: MEG→MEG.61 ± .08.71 ± .11
into a new participant: EEG→MEG.69 ± .09.76 ± .10
from a new participant (nothing fitted): EEG→EEG.56 ± .03.92 ± .02

A few minutes of data adapt the encoder. We fine-tune only the brain side (tokenizer, encoder, visual readout) on part of a new EEG participant's data:

new participant's dataretrieval top-1 ↑translation from them ↑
none.06 ± .03.56 ± .03
5% (≈11 min).33 ± .06.76 ± .04
25% (≈55 min).51 ± .03.85 ± .02
100%.64 ± .01.90 ± .02
reference: participant trained in.59 ± .03.92 ± .02

Result. Translation into a new participant stays close to the trained-in level; 11 minutes of data close about half of the retrieval gap, and with all of their data a new participant matches a trained one. Zero-shot retrieval is hard across this literature (Brain-HIVE: EEG top-1 20.0 unseen vs 75.7 seen).

Weakness 1 (b) and Question 3: Translation involving fMRI

BiSURE unifies what is represented, not how it is measured (Eq. 3): the latent keeps a modality-specific part that each instrument's head reads. Separating the two is what makes the representation useful: what an image evokes is shared across people and instruments, while how it is measured differs, so one latent can pool every instrument's data (letting abundant recordings inform scarce ones) and carry a response from any participant or instrument to another. EEG and MEG latents are close enough to be read directly (EEG→MEG .84 ± .02). The fMRI latent carries a much larger modality-specific part (about 17 times the norm of EEG's and MEG's), which an EEG or MEG latent does not have.

Setting. Into fMRI, we first remove the modality-specific part: a linear regression, fitted only on the source participant's own training data, maps the source latent onto the image route (no fMRI data, no paired recordings). Scored on the 14 concepts shared by the EEG/MEG and fMRI test sets.

translation into fMRI ↑BiSURE latent (no paired data)reference: decode to CLIP, then image route
MEG→fMRI.38 ± .05.35 ± .06
EEG→fMRI.21 ± .16.29 ± .06

Result. Translation into fMRI works without any paired data, on par with decoding to CLIP. Out of fMRI, the latent is read directly by the EEG and MEG heads (fMRI→EEG .19 ± .08, fMRI→MEG .40 ± .23).

Weakness 1 (c): The 84% vs 43% asymmetry

The asymmetry comes from the source, not from the latent. An MEG test response averages 12 different images of a concept, while the EEG target is one image: even the images themselves, averaged per concept and passed through the image route, reach only .80 when read by the EEG heads. MEG is also the data-scarcest modality (four participants, single trials), and its latent has the smaller stimulus-specific part. With EEG as the source, translation reaches .84.

Weakness 1 (d): EEG and MEG share 16,520 training images

Setting. Three retrained variants: disjoint images (EEG keeps 5 of each concept's 10 training images and MEG the other 7 of 12, so no image is shared), disjoint concepts (EEG and MEG train on different halves of the concepts), and a control with the same halved data but shared images.

EEG→MEG translation ↑shared images (control)disjoint imagesdisjoint conceptsno tie
mean ± sd.78 ± .05 (2 seeds).70 ± .07.70 ± .03 (2 seeds)≤ .06

Result. Without a single shared image or concept, translation stays at .70; shared images add only about .08, and the tie provides the rest. The link does not run through fMRI either: a model trained on EEG and MEG alone translates EEG→MEG at .83 ± .02.

Weakness 2 and Question 2: A capacity- and supervision-matched CLIP baseline

Setting. The paper's no-tie model N is exactly this baseline: same architecture, data, CLIP targets and response heads, trained independently without the tie. Its CLIP-bottleneck route decodes the source response to CLIP and predicts the target's response from it (brain → CLIP → g_φ → response head). We compare the correlation of each translated response with the target's measured response.

translation r ↑BiSURE latentN, CLIP bottleneck
EEG→EEG.0906 ± .0003.0767 ± .0015
MEG→MEG.1088 ± .0026.0818 ± .0025
fMRI→fMRI.1946 ± .0022.1565 ± .0024
EEG→MEG.1059 ± .0031.0857 ± .0012
MEG→EEG.0427 ± .0019.0420 ± .0007

Result. BiSURE's latent translates better than the matched CLIP bottleneck in four of five directions and is level on MEG→EEG, so the gain comes from the tied latent rather than from visual alignment alone.

Question 1: Overlap between the splits

Each dataset's split is clean internally. Across datasets:

training set → test setshared test conceptsshared test images
fMRI → EEG87 / 20077 / 200
fMRI → MEG86 / 2001,032 / 2,400
EEG, MEG → fMRI86 / 1000 / 100

Setting. BiSURE retrained with every overlapping concept removed from training (3 seeds).

BiSUREoverlap removed
EEG top-1 ↑.594 ± .009.584 ± .025
MEG top-1 ↑.179 ± .020.176 ± .014
fMRI top-1 ↑.822 ± .027.824 ± .023
EEG→MEG translation ↑.84 ± .02.81 ± .02

Result. All changes are within seed noise, so the results do not depend on the overlap. The revision adds both tables (Appendix D.5).

Question 4: Temperature fitted on the test gallery

Yes, the bits temperature is fitted on the test gallery, for every method; top-k accuracy and encoding do not use it. Fitting it instead by 5-fold cross-fitting over concepts changes bits by at most .023 on the 29 rescored Table 2 rows and leaves the order of methods unchanged. The revision adds this check (Appendix C.1).

Thank you again for the constructive review. Your suggestions have made the evidence for the central claim considerably stronger. We hope these results resolve your concerns; if so, we would be grateful if you would consider raising your score.

I thank the authors for their constructive engagement with my review (and the others below). The added experiments fully addressed my points. I will keep my score at the current level, as I think it accurately describes the paper's impact.

Comments on the paper

2 comments

Thank you for the follow-up and for your willingness to re-read the revised paper.

We agree that the lack of a validation set is not an ideal evaluation protocol, and we acknowledge the possibility of selection bias. We also agree that prior work following the same practice does not make it a good reason. We appreciate your care for the rigor of our work: in our future work we will hold out a validation set from the start.

However, this should be distinguished from extensive hyperparameter tuning over the test set. We did not run a hyperparameter search (no grid, random or automated search). The design choices in the paper follow from the method's motivation, for example the stop-gradient that keeps the brain-derived latent as the reference. Our ablations serve as evidence for these choices. They are small and controlled: each changes one or two settings, while everything else, including the optimizer, stays the same. Before each run, we wrote down the outcome we expected.

To address your concern directly, we repeated the two central choices on MEG test data that never entered any decision. Our MEG split follows BrainFLORA and holds out 200 whole concepts, as THINGS-EEG does. THINGS-MEG also has its own test trials: 200 further images, each shown 12 times, which are not part of that split. None of our models had ever been evaluated on them, so no setting or checkpoint was chosen with them. If our choices only held on the test split we used, they should change here.

Check 1: the encoder architecture. We compared MEG encoders that differ only in depth and in a residual skip connection. The same trained models, unchanged, scored on both test sets (MEG 200-way top-1 ↑):

encodertest split we useduntouched test trials
depth 8, with skip (paper's encoder).161.224
depth 4, no skip.156.211
depth 8, no skip.146.196
depth 4, with skip.144.185
depth 6, with skip.141.196

Result. The two test sets rank the encoders consistently (Spearman ρ\rho = .82, Pearson rr = .93), and both select the encoder used in the paper.

Check 2: the central ablation, the tie and its stop-gradient. Its criterion is translation, so we scored MEG→MEG translation (fraction of the target's own image-driven prediction reached) for the same trained models on both test sets; mean ± sd over 3 seeds:

MEG→MEG translation ↑test split we useduntouched test trials
BiSURE (tie, stop-gradient on ZbrainZ_{\mathrm{brain}}).868 ± .007.630 ± .009
tie without stop-gradient.820 ± .034.596 ± .006
tie with stop-gradient on ZstimZ_{\mathrm{stim}} (1 seed).472.345
no tie.015 ± .022.010 ± .014

Result. The ordering is identical on both test sets, and both select the design in the paper. Translated predictions are as accurate on the untouched trials as on our split (rr = .112 vs .109). The fraction is lower only because the image-driven reference is higher there.

In both checks, the decision is the same whether or not the test split we used is consulted.

Thank you again for holding our work to this standard. We hope these checks resolve your concern, and we would be grateful if you would consider raising your score.

Dear Area Chair,

Thank you for overseeing our submission. We summarize its contributions, the strengths the reviewers recognized, and the new evidence.

1. What BiSURE contributes

BiSURE learns one latent for images and EEG, MEG and fMRI responses that supports both retrieval and response prediction. A single tie term makes the image-derived and brain-derived latents interchangeable:

  • Translation between brains without the stimulus. A participant's response, read through another participant's response head, reaches 87–96% of the image-driven prediction, and EEG reaches MEG at 84%, with nothing fitted for the pair and no paired EEG–MEG recordings. It beats a pair-fitted ridge map in 559 of 564 pair-seed cells.
  • An identified mechanism. Without the tie, retrieval is essentially unchanged but translation falls to at most 6%; the tie also raises encoding by 15–31%.
  • One model at specialist level in both directions. Under a common protocol, retrieval matches or exceeds the strongest specialist on every instrument, and encoding exceeds BERG for all 17 participants.

The reviewers recognized these strengths:

  • MnnC: "a convincing evaluation of representational interchangeability"; "the latent-tie ablation is particularly convincing".
  • F1re: the "consistency-objective ablation and CLIP-route comparisons provide useful evidence that the learned latent supports response translation".
  • 7gYD: "strong performance on three different brain modalities, in both decoding and encoding directions".
  • unRw: the model "effectively builds this shared representation space and enables translating between modalities even without paired data".

2. New evidence from the discussion

  • New participants (MnnC, F1re). Into a held-out participant, with only their own response head and embedding fitted, EEG→EEG translation reaches 82% (trained participants: 90%). From a new EEG participant, fine-tuning only the brain side on 11 minutes of their data raises translation from 56% to 76%; with all their data it reaches 90% (trained participant: 92%).
  • Translation into fMRI (MnnC, F1re). On 87 held-out concepts, EEG→fMRI and MEG→fMRI translation through the decoded-CLIP route reaches 44–65%, level with or above a regression fitted on paired images, without paired data.
  • No shared training images (MnnC, F1re). With EEG and MEG trained on disjoint images or disjoint concepts, EEG→MEG translation is 70% (shared-image control: 78%); without the tie it is at most 6%. A model trained on EEG and MEG alone reaches 83%.
  • Matched CLIP bottleneck (MnnC, F1re). Matched in architecture, data and CLIP targets, BiSURE translates better in 4 of 5 directions, level in the fifth.
  • What the latent carries (F1re). An exact split of the translation correlation shows that 20–55% remains after removing either the image-predicted or the CLIP-decoded response. It is positive in all but a few participant pairs; a shuffled source gives about 0.
  • Combining modalities (F1re). Training jointly with EEG raises MEG encoding by about 50% (rr .083 → .129). With fMRI added, the gain holds and fMRI encoding rises over fMRI alone (.195 → .203).
  • 84% vs 43% asymmetry (MnnC, F1re). It comes from the source: an MEG test response averages 12 images of a concept; even these images, averaged, reach only 80% through the EEG heads.
  • Stop-gradient (7gYD). Without it, the fMRI latent loses its stimulus-specific part (ΔZ\Delta_Z .52 → .09); placing it on ZstimZ_{\mathrm{stim}} lowers encoding in all three modalities and EEG→MEG translation (84% → 58%, 1 seed). Figure 4 adds a UMAP of the latent.
  • Cross-dataset overlap and temperature (MnnC). Removing overlapping concepts leaves every metric within seed noise; a cross-fitted temperature changes bits by ≤ .023 and keeps the method ranking.
  • Hyperparameters (unRw). Validation-selected checkpoints change test top-1 by < .02. On THINGS-MEG's own test trials, never used in any decision, the paper's encoder and tie design are selected again.
  • Recent baselines (unRw). Brain-HIVE and Liu et al. learn their retrieval gallery; with it fixed, Brain-HIVE (our re-run) reaches .439 EEG top-1, against BiSURE's .594.
  • Presentation (unRw). The revision adds architecture figures and defines all symbols at first use.

3. Summary

We believe the discussion has addressed most concerns (F1re: "most of my previous concerns have been resolved"). For the points that remained less clear (interpretation of the shared latent, hyperparameter selection), we added the analyses above.

More broadly, we believe neural signals should have a representation of their own, rather than relying on alignment to large pretrained models. BiSURE learns its latent from brain responses, with CLIP as a readout target rather than the latent itself; this latent translates between participants and instruments and carries what brains share beyond CLIP-based predictions.

Thank you for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.