- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces a multimodal representation learning approach for reconstructing audio (music) from the corresponding EEG signal. Specifically, the proposed approach leverages an EEG encoder that tokenizes each electrode separately into temporal patches, allowing for learning temporal and spatial (across channels) interactions. The encoder is first pre-trained on unlabeled recordings in a DINO-like self-distillation manner, where the pre-text task is masked reconstruction from shorter segments. In turn, this pre-trained EEG encoder is then fine-tuned using contrastive alignment to audio embeddings from a pretrained (and frozen) CLAP audio encoder. The resulting aligned embeddings are then used to condition a (frozen) AudioLDM generator for music reconstruction. Results from experiments on the NMED-T and NMED-H datasets comparing few EEG foundation models and music reconstruction methods show that the proposed approach can lead to some improvements. Additional ablation studies indicate that the largest contribution comes from the pre-training of the encoder and the initial per-electrode tokenization. The authors also provide a theoretical analysis of the effects of preserving electrode information (and the amount of it) on the downstream performance (albeit under a simple linear model), the regularization effects of channel masking, and their downstream implications for music identification.
Critical strengths
-
The paper focuses on an interesting (and challenging) task of reconstructing music from EEG signals.
-
The authors provide several ablation studies, evaluating the contributions of different components.
Critical weaknesses
-
W1. The experimental setup is rather narrow (in terms of datasets) and incomplete (in terms of methods). While the authors evaluate a few EEG foundation models, other related methods are not considered. This is particularly the case for EEG-DINO, which has been mentioned in the related work and shares many similarities with the proposed EEG representation learning mechanism. More generally, to this end, the evaluation considers only NMED-T/H, but there are several potentially relevant datasets (like OpenMIIR and CantiSong). At least the authors should discuss their design decisions in more detail.
-
W2. The evaluation protocol is also concerning with respect to the train/test split criteria. In particular, the authors split randomly 1-second segments from the same recordings, which can lead to temporary adjacent segments (both EEG and/or music) being used for training and testing. Although this is acknowledged, the evaluation doesn't establish whether the proposed approach can generalize well to unseen subjects/music. Previous work (eg., Postolache et al. 2025) has already considered more challenging evaluation scenarios (like held-out samples), so it is unclear why a similar setup wasn't followed here as well.
-
W3. The theoretical analysis, while appreciated, is not really convincing in the context of the presented work. Sec. 4.1 considers an OLS with a fixed projection, which is substantially different from the proposed work and the learnt representations. As such, I don't really see how these findings transfer. Sections 4.2 and 4.3, while more related, provide limited insight into the observed empirical behavior, and it remains unclear how these theoretical findings explain the reported improvements.
Questions for authors
-
See weaknesses
-
Could the authors clarify the different pretraining configurations (in Table 2) and their rationale? Specifically why is CSBrain further pretrained, while the other trained from scratch? Why not the other encoders too? In this context, does CBramMod keep the same 125-22 channel projection as in the previous experiment?
-
Could the authors comment on the channel dropout effects across the two training stages? The ablation removes channel dropout from both pretraining and fine-tuning simultaneously, rather than evaluating its contribution in each stage separately.
-
The paper doesn't provide an analysis of the computational efficiency of the proposed approach (and its components). Considering the computational requirements of EEG foundation models, it would be beneficial to present these analyses, esp. in terms of training (in the two stages) and inference times, model sizes and FLOPs, memory consumption, etc. This would help better contextualize the reported performance wrt the baselines.
-
How much stochasticity in the generation process in AudioLDM affects the reconstruction quality (and evaluation)? Given some of the relatively small differences in generated-audio CLAP similarity between some of the methods, this can have an effect.
-
Are there any potential effects of response latency that can influence the EEG-audio alignment? Would different temporal offsets influence the learnt representations (and downstream performance)?
General comment: I find the current presentation/discussion of the results insufficient. As it stands, the authors largely repeat the results that are already present in the tables, rather than discussing their meaning. I suggest the authors use the space more productively, discuss the motivation behind some of their design choices in more detail, and provide their intuition behind the results.