acceptodds
Under review as a conference paper at ICLR 2027

Visual-to-M/EEG Brain Encoding with Time-Frequency Dual-Path Finite Scalar Quantization

Abstract

Visual brain encoding predicts neural responses from visual stimuli, advancing our understanding of visual perception. It is also an indispensable step in visual prostheses, where predicted neural responses guide electrode stimulation patterns. Prior visual brain encoding research has primarily focused on fMRI, whose limited temporal resolution restricts its applicability. While recent work has shifted toward M/EEG-based brain encoding with diffusion models, these approaches treat M/EEG as images—neglecting their frequency-domain structure—and do not explicitly leverage real neural responses in the training objective. We propose a three-stage visual brain encoding framework. In Stage 1, we propose a novel VAE, called BrainVAE, with a time-frequency dual-path encoder: the input M/EEG signal is treated as a 2D image (channels time) and processed by a Conv2d backbone to extract local spatial-temporal features, while simultaneously transformed by FFT to obtain power spectra. The Conv2d feature tokens and Frequency Prefix Tokens are fused by a Transformer via self-attention, incorporating explicit frequency-domain priors. Finite Scalar Quantization (FSQ) discretizes the latent space without codebook collapse or auxiliary losses. In Stage 2, InfoNCE contrastive learning aligns the quantized neural latent space with I-JEPA image embeddings. In Stage 3, a Transformer-based BrainTokenRegressor predicts FSQ token sequences from I-JEPA image embeddings via cross-attention, with losses computed directly against real neural responses in both token and signal spaces. Experiments on THINGS-EEG2 and THINGS-MEG demonstrate the effectiveness of our approach.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain encoding, visual prostheses, MEG, EEG, modality alignment, frequency domain

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_f7U1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies visual-to-M/EEG encoding: predicting neural responses from visual images. It proposes a three-stage framework. First, a neural autoencoder combines Conv2d-derived spatiotemporal tokens with FFT power-spectrum prefix tokens through a Transformer and applies finite scalar quantization (FSQ). Second, global InfoNCE alignment relates the neural representation to frozen I-JEPA features while retaining reconstruction supervision. Third, a cross-attention BrainTokenRegressor predicts neural proxy tokens, and a jointly fine-tuned decoder reconstructs waveforms using token-space and signal-space MSE losses. Experiments cover THINGS-EEG2 and THINGS-MEG, including within-subject, cross-subject, and leave-one-subject-out evaluation. The paper reports within-subject PCC values of 0.666 and 0.558, respectively, and provides frequency-band, temporal-window, sensor-region, and image-encoder analyses.

Critical strengths
  1. A coherent practical formulation of visual neural-response prediction. Separating neural representation learning, cross-modal alignment, and image-conditioned prediction is understandable. Frequency-prefix fusion and combined token/waveform supervision are concrete modeling choices that could be useful for this task. The potential value of the framework is therefore worth investigating, independently of whether each constituent module is new.
  2. Evaluation spans two modalities and transfer settings. The EEG and MEG experiments, cross-subject/LOSO evaluations, and Stage 2/loss ablations provide useful initial evidence. In Table 2, removing Stage 2 reduces the reported PCC from 0.666 to 0.606 on EEG and from 0.558 to 0.515 on MEG. The supplied scripts also clarify several implementation details, including regressor defaults and checkpoint-selection criteria. These positive aspects do not, however, resolve the empirical and methodological concerns below.
Critical weaknesses

1. The empirical results are not internally reconciled, and the supplied checkpoint/evaluation code does not establish reproducible, independently held-out performance.

Numerical consistency. Table 5 (Appendix A.4, pp. 15-16) is inconsistent with an ordinary MSE decomposition under a common evaluation protocol. For identical predictions, targets, signal scale, and sample weights, a disjoint channel partition must satisfy MSEall=∑rCrCMSEr.\mathrm{MSE}_{\mathrm{all}}=\sum_r\frac{C_r}{C}\mathrm{MSE}_r. The reported EEG channel counts (15,21,14,10,3) and regional MSE values (0.081,0.030,0.041,0.058,0.075) imply approximately 0.05117, not the reported 0.021. For MEG, the counts (95,16,65,21,74) and values (0.534,0.459,0.576,0.487,0.598) imply approximately 0.55348, not 0.433. Both reported overall errors are below every regional error. Table 4 (p. 9) has a related MEG discrepancy: the stated nonoverlapping temporal windows imply 0.3*0.289 + 0.2*0.397 + 0.5*0.517 = 0.4246, whereas the full-window MSE is 0.433. These differences exceed the displayed rounding. Different normalization, samples, checkpoints, or averaging schemes could make the identity inapplicable, but such differences must be disclosed. This check applies to MSE, not PCC. A common prediction/target output and the exact table-generation code are needed to resolve the issue.

Incomplete checkpointing of the stated main model. Section 3.4 and Appendix A.2 specify joint decoder fine-tuning in Stage 3. Consistently, train_regressor.py makes vae.decoder trainable and adds it to the optimizer when --unfreeze_decoder is enabled (lines 384-389 and 450-458). However, the best-checkpoint branch saves only the regressor's state_dict() (lines 530-538), with no corresponding save of the updated decoder in this script. Consequently, the saved regressor plus the original Stage 2 VAE does not reconstruct the complete model evaluated at the selected epoch. The matched decoder parameters and buffers are required. This is a concrete reproducibility gap in the inspected script, not evidence that its in-memory measurements were fabricated; an alternative saving/evaluation pipeline should be identified if one produced the reported results.

Final-test independence and response definition remain unresolved. Both training scripts select checkpoints by the highest correlation on val_dl. The regressor stores the selected epoch's metrics as best_results and saves them to results.npy; the adjacent comment describes these as test-set metrics for subsequent aggregation (lines 530-565). There is no separate post-selection test evaluation in the inspected scripts. This is legitimate if val_dl is an independent validation split and a separate final test is evaluated elsewhere; it is not an untouched-test protocol if val_dl is the official test set and these peak-selected metrics populate Table 1. The loader and aggregation script are needed to distinguish these cases. I am not inferring leakage from the variable name or comment alone. Additionally, both preprocessing functions average dimension 1 of four-dimensional EEG inputs (train_vae.py, lines 103-114; train_regressor.py, lines 129-142). Whether this averages repeated presentations or merely removes a singleton dimension is unresolved without the loader. This affects the distinction between single-trial and mean-response prediction and the comparability of baselines.

Metric aggregation and distributed execution need verification. train_regressor.py averages batch-mean MSE values equally (lines 281-303), which differs from sample-weighted MSE when batch sizes differ. Both validation loops retain local metric lists without a visible cross-process gather/reduction, although the validation loaders pass through accelerator.prepare. Under sharded multi-process evaluation, those metrics describe a local shard rather than the complete evaluation set. Stage 3 also invokes the trainable decoder through accelerator.unwrap_model(vae).decode(...) (lines 174-192), so synchronization through the actual distributed wrapper needs verification. These concerns depend on the launcher: four GPUs could instead have been used for independent single-process jobs. They should not be treated as established failures of every reported experiment. Nor do they, by themselves, explain the table discrepancies. The actual launch configuration and a checkpoint-reload evaluation are needed.

Taken together, these issues prevent the large reported gains from being used as reliable evidence without reconciliation. The main comparisons also need explicit baseline provenance and matched signal scales, repetition handling, and sample-aggregation rules; a deterministic MSE predictor and stochastic generators should be compared against clearly defined, equivalent targets.

2. The distinctive frequency-aware and discrete mechanisms are not isolated, and the validation does not yet establish their claimed advantage over simpler prediction models.

Table 2 ablates Stage 2 and the two Stage 3 losses, but not the two mechanisms emphasized in the title: frequency-prefix fusion and FSQ. A matched no-prefix counterpart and a continuous-latent counterpart would directly test whether these mechanisms add value beyond the rest of the architecture and training procedure. A simple regressor using the same visual features would help assess the staged framework. The code exposes a frequency-prefix switch, alternative VAE paths, and perplexity logging, but these interfaces are not themselves experimental evidence. Likewise, post-hoc frequency-band accuracy does not establish the causal contribution of the frequency pathway. My novelty concern is therefore not simply that known modules are reused; it is that the specific combination has not been distinguished empirically from simpler alternatives.

The role of discreteness also requires clarification. Section 3.5 specifies s^=G(R(Ψ(I)))\hat{s}=\mathcal G(\mathcal R(\Psi(I))), and the visible code directly decodes predicted proxy tokens, without an explicit quantizer at the call site. Quantization inside decode cannot be checked without vae.py. FSQ may be valuable as a training-target constraint even if prediction is continuous, but the current experiments do not isolate that value from autoencoder pretraining, token supervision, and decoder adaptation. Similarly, the stated inference path specifies no random input; this is a valid point-prediction setting, but does not by itself demonstrate trial-distribution modeling. The BrainVAE terminology should be accompanied by its intended probabilistic interpretation rather than used to imply capabilities not evaluated.

The exact reported configuration also needs to be traceable. The regressor defaults to CLIP, and frequency prefixes, signal-space loss, and decoder unfreezing are opt-in. These defaults do not prove that the experiments used an incorrect configuration, since command-line overrides can implement the paper. However, the actual per-experiment commands and matched checkpoints are essential. There is also a concrete Stage 2 discrepancy: _train_epoch_stage2_full applies a learned signal_proj to the mean-pooled neural tokens (train_vae.py, lines 346-371 and 525-537), whereas Section 3.3 describes alignment directly with the pooled neural representation. This should be reconciled with the equations.

The motivation contrasting direct supervision with diffusion is too strong (Sections 1-2). As a technical comparison with Ho et al. (2020, Eqs. 4, 11, and 14), noise prediction is linked to clean-signal estimation through the forward process and its timestep-dependent weighting; it does not constitute an absence of supervision from real data. Direct waveform regression may better suit the chosen point-estimation metrics, but that advantage must be established under matched evaluation rather than inferred solely from how the objective is written.

Finally, flattened waveform PCC and MSE do not isolate image-specific information from a common evoked-response pattern. A training-derived mean-response baseline and mismatched-image conditioning would directly test whether the model predicts differences between stimuli. The headline comparisons also lack subject- and training-seed-level uncertainty sufficient to assess the stability of the gains: signal_corr_std in the supplied script is the spread across evaluated examples, not uncertainty across subjects or independent training runs. The biological interpretation should remain correspondingly limited. Figure 5 averages real training signals rather than comparing measured and predicted topographies, and the experiments do not validate prosthetic stimulation or perceptual outcomes. I am not requesting a new clinical study; these broader applications should remain motivations rather than established capabilities.

Overall assessment. I recommend rejection in the current form. The decisive concerns are the unresolved empirical consistency and reproducibility issues, together with insufficient evidence for the particular methodological contribution. This is not an allegation of fabrication or confirmed test leakage. A reconciled evaluation using correctly paired checkpoints and independent data, together with focused controls of the existing claims, could materially change the assessment.

Questions for authors

1. Numerical reconciliation and checkpoint replay (clarification). Can Tables 4-5 be recomputed from the same saved prediction/target tensors, with explicit normalization, masks, sample weights, and checkpoint IDs? Please explain the MSE discrepancies and identify any changes to Table 1. How is the fine-tuned decoder at the selected epoch saved, and does reloading it with the selected regressor reproduce the reported metrics?

2. Data isolation and repetition handling (clarification). What files and image/concept/subject IDs populate train_dl and val_dl in each stage? Is val_dl an independent validation split or the official test partition? What does the averaged dimension of four-dimensional EEG inputs represent, and how many repetitions are used? Please trace how the selected results.npy produces the final tables and identify the separate final-test evaluation, if present. Appendix A.1 does not by itself imply that the other subjects' test data were used for training.

3. Actual implementation and execution (clarification). Please identify the data/model modules, actual launch commands, and configurations corresponding to the main results. Were the four GPUs used in one data-parallel job or independent jobs? If data-parallel, where are full-dataset metrics collected and how is the jointly trained decoder synchronized despite the unwrapped call? Please also reconcile the additional neural projection in Stage 2 and specify whether predicted tokens are quantized inside decode.

4. Evidence for the headline components (targeted validation). Are matched no-frequency-prefix and continuous-latent controls available under the same training, capacity, and model-selection protocol? Please provide or point to these results and a same-visual-feature simple regression baseline. These controls would determine whether the contribution comes specifically from the proposed frequency/discrete design rather than the surrounding pretraining and supervision.

5. Stimulus specificity and comparison uncertainty (targeted validation). How do predictions compare with a training-derived mean response and with mismatched image conditions, with the trained model otherwise fixed? Please provide paired subject-level effects and variability across independent training runs for the principal comparisons and core ablations. Which external baselines were rerun versus quoted, and are response averaging, signal scale, and stochastic-sample aggregation matched? These answers would materially affect interpretation of the gains.

Reviewer_aoaoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a three-stage framework for visual-to-M/EEG brain encoding. In Stage 1, a BrainVAE autoencoder learns to reconstruct M/EEG signals. In Stage 2, the BrainVAE is fine-tuned with an InfoNCE loss that aligns its latents with frozen I-JEPA image embeddings. In Stage 3, a Transformer regressor maps I-JEPA embeddings to the BrainVAE latents, trained with both a latent-space MSE and a signal-space MSE against the real neural responses. On THINGS-EEG2 and THINGS-MEG, the authors report a strong performance against some recent baselines. They also include cross-subject and leave-one-subject-out alongside additional ablations.

Critical strengths

While I have some doubts that all experimental results are correctly documented in the manuscript, as discussed in the weaknesses section, if the results are to be taken at face value, the proposed method seems to be much better than previous encoding models, for both EEG and MEG. Ablation studies in section 4.2 support that stage 2 training as well as the training objectives are useful. Swapping image encoders in section 4.6 also show that the performance is somewhat stable across different image encoders, showing the gain is not because the proposed method simply used a more advanced image encoder.

Critical weaknesses
  1. Many experimental results look strange. In table 1 EEG results, the MSE of the proposed method seems extremely lower than the other baselines (more than 5x lower than the best baseline). Also for EEG, the cross-subject MSE is comparable to the other two LOSO MSE, while the PCC and CosSim is much worse. The trend looks more reasonable for the MEG results; the cross-subject is much worse than the two LOSO in all three metrics. Also comparing the LOSO results with some baselines, the two LOSO results have better PCC and CosSim than Xu et al. yet they have much worse MSE (about 3.5x). The authors would need to double check if these results are accurate and if so provide some analysis on the numbers to explain why this is the case. On a similar note, I am assuming the authors pulled some or all of the baseline numbers from existing work: Güçlü, Yamins, SynBrain and Xu et al. seem to be copied from the Xu et al. (2025) paper, and the other two rows seemed to be copied from an ArXiv preprint not cited here (Xu, Ganxi, et al. "ViBE: Visual-to-M/EEG Brain Encoding via Spatio-Temporal VAE and Distribution-Aligned Projection." arXiv preprint arXiv:2604.26218 (2026)). The authors would also have to double-check if all models are trained and evaluated using the exactly same data, and there are no confounds due to different normalization or such during evaluation. Beyond table 1, some table 3 results also look strange. The MSE for both EEG and MEG is much larger than the MSE of all bands combined: the full-band MSE (0.021 EEG / 0.433 MEG) is more than twice the sum of the five band-wise MSEs (0.009 / 0.215). Also for MEG, PCC and CosSim for the full-band is larger than the best individual band. Both of those points seem unintuitive for me, and I would like an explanation for these results.

  2. Some key ablation and analysis about FSQ are absent. The introduction of FSQ, listed as one of the contributions by the authors, needs more explanation and/or ablation on why discretization of the latent space is needed for the brain encoding pipeline. Although the authors explain why they used FSQ over traditional vector quantization, they still need to justify why such quantization is needed in the first place. One way to show that would be performing ablation on the FSQ module by removing it completely (therefore using no bottleneck in the pipeline) or replacing with a continuous bottleneck.

  3. The proposed model cannot differentiate different subjects, and therefore cannot learn from other subject's data, and also performs poorly in multi-subject settings. Some existing brain encoders (Beliy, Roman, et al. "The wisdom of a crowd of brains: A universal brain encoder." arXiv preprint arXiv:2406.12179 (2024).) allow the brain encoder to "remember" which subject it is encoding the information for, and leverages this ability to create appropriate neural signals for different subject, and also allows cross-subject knowledge transfer by training with data from multiple subjects. SynBrain also allows cross-subject training in the paper. Currently the proposed model seems to not benefit from cross-subject knowledge transfer but rather be harmed by it, if we compare the regular single-subject model with the LOSO within-subject model from Table 1. This is to be expected, as in the LOSO within-subject setting, the model is trained with different neural signals from the same image (trained with multiple outputs for one input), and without a way to differentiate between different subjects, data from other subjects will only act as noise rather than containing useful information. In that sense, the analysis surrounding the cross-subject and LOSO experiments would need more work, and I have to disagree with some of the points the authors make. The biggest one is how the task is formulated and what we could expect from the model. Since the model cannot differentiate between different subjects, the task would be about generating neural responses without any prior information of the specific subject. In the cross-subject setting, the model is asked to generate the neural response for a random subject it never saw before, which should be virtually impossible to perfectly solve. In the LOSO within-subject setting, if I understood the setting correctly, the model is asked to generate the neural response for all the different training subjects while only given the image as the input, which should be literally impossible since it has to generate different signals for each subject from one same input. In these settings the best a model can do is to generate the signal from the train subject (when trained with one subject) or the population average (when trained with multiple subjects). The difference between the response for the (train / test subject) or the (population average / test subject) would thus be the performance ceiling of the task, and the authors should compute and report these statistics to see how much the proposed model approaches this performance ceiling. Considering this, I would attribute the small difference between the LOSO within-subject and LOSO cross-subject results to this collapse to the population average, rather than "accomodating inter-subject variability" as the authors claim in section 4.1.

Questions for authors
  • As discussed in weakness #1, are the results in table 1 and table 3 accurate? If so, are there any possible explanations for the seemingly inconsistent results?
  • Why does the proposed method use a discrete quantization scheme? Can it use a continuous method instead?
  • As discussed in weakness #3, what is the population-level performance ceiling for the cross-subject and LOSO within-subject experiments? How close is the proposed method to this ceiling?
  • Can other baselines be applied to the single-subject cross-subject and the LOSO experiments? Do the authors have any results for that?
  • Does the proposed method have a straightforward way to be modified to better learn from multiple subjects?
Reviewer_qimaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper predicts M/EEG responses to visual stimuli (visual brain encoding), motivated by visual prostheses. Its starting point is a critique of Xu et al. (2025), the only prior M/EEG encoding work: their DiT treats signals as 2D images and ignores frequency structure, and the diffusion objective never explicitly supervises predictions against real neural responses. The authors replace this with a three-stage pipeline: a BrainVAE whose encoder mixes Conv2d features with FFT power-spectrum "prefix tokens" (discarded after fusion, so frequency priors cost no latent dimensions), discretized via FSQ; an InfoNCE stage aligning neural latents with I-JEPA embeddings; and a token regressor trained against real responses in both token and signal space. On THINGS-EEG2 and THINGS-MEG, PCC improves from 0.425 to 0.666 and from 0.379 to 0.558 over the strongest baseline under the same protocol. Ablations, band/time-window/ region analyses, and LOSO generalization experiments are included.

This is a solid, focused engineering paper with a real motivation and convincing controlled gains. My main reservation is that its headline contribution is not isolated by an ablation.

Critical strengths
  1. The critique it builds on is specific and fair, and the design follows from it stage by stage. The gains under an identical protocol (0.425 to 0.666 PCC) are large, and the token-only ablation (PCC drops to 0.381) backs up the claim that direct signal-space supervision matters. This is not a paper where you have to guess why the method should work.

  2. The analysis section does more than fill space. Band-wise results, early time windows dominating, occipital channels reconstructing best (PCC 0.758), and the honest cross-subject numbers (0.223 single-source, 0.479 under LOSO) are all consistent with known visual neuroscience and easy to check against the tables.

Critical weaknesses
  1. Table 2 removes Stage 2 and varies the Stage 3 losses, but nothing removes the frequency prefix tokens. As written, the paper cannot tell us whether the dual path earns its place or whether most of the gain comes from swapping the diffusion objective for latent regression. For a paper named after this module, that control has to be in the paper, not left to the reader's guess.

  2. The work stays very close to Xu et al. (2025): same datasets, same preprocessing, same evaluation. The deltas are real but engineering-level, and there is no matched-capacity or matched-budget comparison to tell how much of the gain is architectural versus just an easier training objective. The prosthesis motivation is fine as long-term context, but predicting scalp EEG to natural images is not yet stimulation patterns, and the paper would read more honestly without leaning on it.

Questions for authors
  1. Could you add the missing control including the full pipeline without the frequency prefix tokens, on both datasets? This would settle whether the dual path itself drives the improvement.

  2. How do the results hold up against the DiT baseline at matched training budget and model size? Without this it is hard to attribute the gain to the design rather than to regression being easier to train than diffusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.