- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies visual-to-M/EEG encoding: predicting neural responses from visual images. It proposes a three-stage framework. First, a neural autoencoder combines Conv2d-derived spatiotemporal tokens with FFT power-spectrum prefix tokens through a Transformer and applies finite scalar quantization (FSQ). Second, global InfoNCE alignment relates the neural representation to frozen I-JEPA features while retaining reconstruction supervision. Third, a cross-attention BrainTokenRegressor predicts neural proxy tokens, and a jointly fine-tuned decoder reconstructs waveforms using token-space and signal-space MSE losses. Experiments cover THINGS-EEG2 and THINGS-MEG, including within-subject, cross-subject, and leave-one-subject-out evaluation. The paper reports within-subject PCC values of 0.666 and 0.558, respectively, and provides frequency-band, temporal-window, sensor-region, and image-encoder analyses.
Critical strengths
- A coherent practical formulation of visual neural-response prediction. Separating neural representation learning, cross-modal alignment, and image-conditioned prediction is understandable. Frequency-prefix fusion and combined token/waveform supervision are concrete modeling choices that could be useful for this task. The potential value of the framework is therefore worth investigating, independently of whether each constituent module is new.
- Evaluation spans two modalities and transfer settings. The EEG and MEG experiments, cross-subject/LOSO evaluations, and Stage 2/loss ablations provide useful initial evidence. In Table 2, removing Stage 2 reduces the reported PCC from 0.666 to 0.606 on EEG and from 0.558 to 0.515 on MEG. The supplied scripts also clarify several implementation details, including regressor defaults and checkpoint-selection criteria. These positive aspects do not, however, resolve the empirical and methodological concerns below.
Critical weaknesses
1. The empirical results are not internally reconciled, and the supplied checkpoint/evaluation code does not establish reproducible, independently held-out performance.
Numerical consistency. Table 5 (Appendix A.4, pp. 15-16) is inconsistent with an ordinary MSE decomposition under a common evaluation protocol. For identical predictions, targets, signal scale, and sample weights, a disjoint channel partition must satisfy
The reported EEG channel counts (15,21,14,10,3) and regional MSE values (0.081,0.030,0.041,0.058,0.075) imply approximately 0.05117, not the reported 0.021. For MEG, the counts (95,16,65,21,74) and values (0.534,0.459,0.576,0.487,0.598) imply approximately 0.55348, not 0.433. Both reported overall errors are below every regional error. Table 4 (p. 9) has a related MEG discrepancy: the stated nonoverlapping temporal windows imply 0.3*0.289 + 0.2*0.397 + 0.5*0.517 = 0.4246, whereas the full-window MSE is 0.433. These differences exceed the displayed rounding. Different normalization, samples, checkpoints, or averaging schemes could make the identity inapplicable, but such differences must be disclosed. This check applies to MSE, not PCC. A common prediction/target output and the exact table-generation code are needed to resolve the issue.
Incomplete checkpointing of the stated main model. Section 3.4 and Appendix A.2 specify joint decoder fine-tuning in Stage 3. Consistently, train_regressor.py makes vae.decoder trainable and adds it to the optimizer when --unfreeze_decoder is enabled (lines 384-389 and 450-458). However, the best-checkpoint branch saves only the regressor's state_dict() (lines 530-538), with no corresponding save of the updated decoder in this script. Consequently, the saved regressor plus the original Stage 2 VAE does not reconstruct the complete model evaluated at the selected epoch. The matched decoder parameters and buffers are required. This is a concrete reproducibility gap in the inspected script, not evidence that its in-memory measurements were fabricated; an alternative saving/evaluation pipeline should be identified if one produced the reported results.
Final-test independence and response definition remain unresolved. Both training scripts select checkpoints by the highest correlation on val_dl. The regressor stores the selected epoch's metrics as best_results and saves them to results.npy; the adjacent comment describes these as test-set metrics for subsequent aggregation (lines 530-565). There is no separate post-selection test evaluation in the inspected scripts. This is legitimate if val_dl is an independent validation split and a separate final test is evaluated elsewhere; it is not an untouched-test protocol if val_dl is the official test set and these peak-selected metrics populate Table 1. The loader and aggregation script are needed to distinguish these cases. I am not inferring leakage from the variable name or comment alone. Additionally, both preprocessing functions average dimension 1 of four-dimensional EEG inputs (train_vae.py, lines 103-114; train_regressor.py, lines 129-142). Whether this averages repeated presentations or merely removes a singleton dimension is unresolved without the loader. This affects the distinction between single-trial and mean-response prediction and the comparability of baselines.
Metric aggregation and distributed execution need verification. train_regressor.py averages batch-mean MSE values equally (lines 281-303), which differs from sample-weighted MSE when batch sizes differ. Both validation loops retain local metric lists without a visible cross-process gather/reduction, although the validation loaders pass through accelerator.prepare. Under sharded multi-process evaluation, those metrics describe a local shard rather than the complete evaluation set. Stage 3 also invokes the trainable decoder through accelerator.unwrap_model(vae).decode(...) (lines 174-192), so synchronization through the actual distributed wrapper needs verification. These concerns depend on the launcher: four GPUs could instead have been used for independent single-process jobs. They should not be treated as established failures of every reported experiment. Nor do they, by themselves, explain the table discrepancies. The actual launch configuration and a checkpoint-reload evaluation are needed.
Taken together, these issues prevent the large reported gains from being used as reliable evidence without reconciliation. The main comparisons also need explicit baseline provenance and matched signal scales, repetition handling, and sample-aggregation rules; a deterministic MSE predictor and stochastic generators should be compared against clearly defined, equivalent targets.
2. The distinctive frequency-aware and discrete mechanisms are not isolated, and the validation does not yet establish their claimed advantage over simpler prediction models.
Table 2 ablates Stage 2 and the two Stage 3 losses, but not the two mechanisms emphasized in the title: frequency-prefix fusion and FSQ. A matched no-prefix counterpart and a continuous-latent counterpart would directly test whether these mechanisms add value beyond the rest of the architecture and training procedure. A simple regressor using the same visual features would help assess the staged framework. The code exposes a frequency-prefix switch, alternative VAE paths, and perplexity logging, but these interfaces are not themselves experimental evidence. Likewise, post-hoc frequency-band accuracy does not establish the causal contribution of the frequency pathway. My novelty concern is therefore not simply that known modules are reused; it is that the specific combination has not been distinguished empirically from simpler alternatives.
The role of discreteness also requires clarification. Section 3.5 specifies , and the visible code directly decodes predicted proxy tokens, without an explicit quantizer at the call site. Quantization inside decode cannot be checked without vae.py. FSQ may be valuable as a training-target constraint even if prediction is continuous, but the current experiments do not isolate that value from autoencoder pretraining, token supervision, and decoder adaptation. Similarly, the stated inference path specifies no random input; this is a valid point-prediction setting, but does not by itself demonstrate trial-distribution modeling. The BrainVAE terminology should be accompanied by its intended probabilistic interpretation rather than used to imply capabilities not evaluated.
The exact reported configuration also needs to be traceable. The regressor defaults to CLIP, and frequency prefixes, signal-space loss, and decoder unfreezing are opt-in. These defaults do not prove that the experiments used an incorrect configuration, since command-line overrides can implement the paper. However, the actual per-experiment commands and matched checkpoints are essential. There is also a concrete Stage 2 discrepancy: _train_epoch_stage2_full applies a learned signal_proj to the mean-pooled neural tokens (train_vae.py, lines 346-371 and 525-537), whereas Section 3.3 describes alignment directly with the pooled neural representation. This should be reconciled with the equations.
The motivation contrasting direct supervision with diffusion is too strong (Sections 1-2). As a technical comparison with Ho et al. (2020, Eqs. 4, 11, and 14), noise prediction is linked to clean-signal estimation through the forward process and its timestep-dependent weighting; it does not constitute an absence of supervision from real data. Direct waveform regression may better suit the chosen point-estimation metrics, but that advantage must be established under matched evaluation rather than inferred solely from how the objective is written.
Finally, flattened waveform PCC and MSE do not isolate image-specific information from a common evoked-response pattern. A training-derived mean-response baseline and mismatched-image conditioning would directly test whether the model predicts differences between stimuli. The headline comparisons also lack subject- and training-seed-level uncertainty sufficient to assess the stability of the gains: signal_corr_std in the supplied script is the spread across evaluated examples, not uncertainty across subjects or independent training runs. The biological interpretation should remain correspondingly limited. Figure 5 averages real training signals rather than comparing measured and predicted topographies, and the experiments do not validate prosthetic stimulation or perceptual outcomes. I am not requesting a new clinical study; these broader applications should remain motivations rather than established capabilities.
Overall assessment. I recommend rejection in the current form. The decisive concerns are the unresolved empirical consistency and reproducibility issues, together with insufficient evidence for the particular methodological contribution. This is not an allegation of fabrication or confirmed test leakage. A reconciled evaluation using correctly paired checkpoints and independent data, together with focused controls of the existing claims, could materially change the assessment.
Questions for authors
1. Numerical reconciliation and checkpoint replay (clarification). Can Tables 4-5 be recomputed from the same saved prediction/target tensors, with explicit normalization, masks, sample weights, and checkpoint IDs? Please explain the MSE discrepancies and identify any changes to Table 1. How is the fine-tuned decoder at the selected epoch saved, and does reloading it with the selected regressor reproduce the reported metrics?
2. Data isolation and repetition handling (clarification). What files and image/concept/subject IDs populate train_dl and val_dl in each stage? Is val_dl an independent validation split or the official test partition? What does the averaged dimension of four-dimensional EEG inputs represent, and how many repetitions are used? Please trace how the selected results.npy produces the final tables and identify the separate final-test evaluation, if present. Appendix A.1 does not by itself imply that the other subjects' test data were used for training.
3. Actual implementation and execution (clarification). Please identify the data/model modules, actual launch commands, and configurations corresponding to the main results. Were the four GPUs used in one data-parallel job or independent jobs? If data-parallel, where are full-dataset metrics collected and how is the jointly trained decoder synchronized despite the unwrapped call? Please also reconcile the additional neural projection in Stage 2 and specify whether predicted tokens are quantized inside decode.
4. Evidence for the headline components (targeted validation). Are matched no-frequency-prefix and continuous-latent controls available under the same training, capacity, and model-selection protocol? Please provide or point to these results and a same-visual-feature simple regression baseline. These controls would determine whether the contribution comes specifically from the proposed frequency/discrete design rather than the surrounding pretraining and supervision.
5. Stimulus specificity and comparison uncertainty (targeted validation). How do predictions compare with a training-derived mean response and with mismatched image conditions, with the trained model otherwise fixed? Please provide paired subject-level effects and variability across independent training runs for the principal comparisons and core ablations. Which external baselines were rerun versus quoted, and are response averaging, signal scale, and stochastic-sample aggregation matched? These answers would materially affect interpretation of the gains.