acceptodds
Under review as a conference paper at ICLR 2027

Channel-Preserving Representation Alignment for EEG-to-Music Reconstruction

Abstract

Reconstructing naturalistic music from EEG requires extracting music-specific information from weak signals distributed across electrodes. Existing spectrogram regression and diffusion approaches provide audio synthesis mechanisms, but learning the EEG-to-music mapping faces an information–estimation tradeoff. Combining electrode measurements into fewer features can ease estimation from limited recordings while obscuring distinctions between music segments. We propose channel-preserving representation alignment, separating electrode information retention from prediction regularization. Per-electrode tokenization preserves spatial identity for learned aggregation, while multi-view self-distillation and channel dropout encourage consistency across electrode subsets during pretraining and music alignment. The aligned representation conditions a pretrained audio generator. Our theory explains these complementary choices. A finite-sample regression analysis identifies when retained task information outweighs estimation cost. For nonlinear alignment, an exact decomposition reveals a prediction-consistency penalty in the masked contrastive objective, with margin-dependent bounds connecting prediction variability to ranking errors across electrode subsets. On NMED-T/H, our method improves within-subject 50-way identification from 0.388 to 0.485 and generated-audio CLAP similarity from 0.635 to 0.683 compared with previous work.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain decoding, neural coding, brain-computer interface

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sB1VOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a multimodal representation learning approach for reconstructing audio (music) from the corresponding EEG signal. Specifically, the proposed approach leverages an EEG encoder that tokenizes each electrode separately into temporal patches, allowing for learning temporal and spatial (across channels) interactions. The encoder is first pre-trained on unlabeled recordings in a DINO-like self-distillation manner, where the pre-text task is masked reconstruction from shorter segments. In turn, this pre-trained EEG encoder is then fine-tuned using contrastive alignment to audio embeddings from a pretrained (and frozen) CLAP audio encoder. The resulting aligned embeddings are then used to condition a (frozen) AudioLDM generator for music reconstruction. Results from experiments on the NMED-T and NMED-H datasets comparing few EEG foundation models and music reconstruction methods show that the proposed approach can lead to some improvements. Additional ablation studies indicate that the largest contribution comes from the pre-training of the encoder and the initial per-electrode tokenization. The authors also provide a theoretical analysis of the effects of preserving electrode information (and the amount of it) on the downstream performance (albeit under a simple linear model), the regularization effects of channel masking, and their downstream implications for music identification.

Critical strengths
  • The paper focuses on an interesting (and challenging) task of reconstructing music from EEG signals.

  • The authors provide several ablation studies, evaluating the contributions of different components.

Critical weaknesses
  • W1. The experimental setup is rather narrow (in terms of datasets) and incomplete (in terms of methods). While the authors evaluate a few EEG foundation models, other related methods are not considered. This is particularly the case for EEG-DINO, which has been mentioned in the related work and shares many similarities with the proposed EEG representation learning mechanism. More generally, to this end, the evaluation considers only NMED-T/H, but there are several potentially relevant datasets (like OpenMIIR and CantiSong). At least the authors should discuss their design decisions in more detail.

  • W2. The evaluation protocol is also concerning with respect to the train/test split criteria. In particular, the authors split randomly 1-second segments from the same recordings, which can lead to temporary adjacent segments (both EEG and/or music) being used for training and testing. Although this is acknowledged, the evaluation doesn't establish whether the proposed approach can generalize well to unseen subjects/music. Previous work (eg., Postolache et al. 2025) has already considered more challenging evaluation scenarios (like held-out samples), so it is unclear why a similar setup wasn't followed here as well.

  • W3. The theoretical analysis, while appreciated, is not really convincing in the context of the presented work. Sec. 4.1 considers an OLS with a fixed projection, which is substantially different from the proposed work and the learnt representations. As such, I don't really see how these findings transfer. Sections 4.2 and 4.3, while more related, provide limited insight into the observed empirical behavior, and it remains unclear how these theoretical findings explain the reported improvements.

Questions for authors
  • See weaknesses

  • Could the authors clarify the different pretraining configurations (in Table 2) and their rationale? Specifically why is CSBrain further pretrained, while the other trained from scratch? Why not the other encoders too? In this context, does CBramMod keep the same 125-22 channel projection as in the previous experiment?

  • Could the authors comment on the channel dropout effects across the two training stages? The ablation removes channel dropout from both pretraining and fine-tuning simultaneously, rather than evaluating its contribution in each stage separately.

  • The paper doesn't provide an analysis of the computational efficiency of the proposed approach (and its components). Considering the computational requirements of EEG foundation models, it would be beneficial to present these analyses, esp. in terms of training (in the two stages) and inference times, model sizes and FLOPs, memory consumption, etc. This would help better contextualize the reported performance wrt the baselines.

  • How much stochasticity in the generation process in AudioLDM affects the reconstruction quality (and evaluation)? Given some of the relatively small differences in generated-audio CLAP similarity between some of the methods, this can have an effect.

  • Are there any potential effects of response latency that can influence the EEG-audio alignment? Would different temporal offsets influence the learnt representations (and downstream performance)?

General comment: I find the current presentation/discussion of the results insufficient. As it stands, the authors largely repeat the results that are already present in the tables, rather than discussing their meaning. I suggest the authors use the space more productively, discuss the motivation behind some of their design choices in more detail, and provide their intuition behind the results.

Reviewer_r9SvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The goal of this paper is to predict music from EEG data. To this end, each electrode signal is tokenized separately, and an encoder is learned using self-distillation with random electrode dropout. This EEG representation is then aligned with the frozen CLAP audio embedding space and used to prompt a frozen AudioLDM generator. The paper includes a theoretical section discussing how information loss and electrode masking affect ranking errors. Results are reported on NMED-T and NMED-H.

Critical strengths

First, the encoder comparisons are controlled, in particular the matched comparison with CBraMod and the control that restricts pretraining to training EEG, although the identical results reported for these two experiments need to be explained. Second, the decomposition of the masked contrastive loss into a mean-prediction term and a consistency penalty is correct and offers a clear interpretation of electrode dropout. The paper is also clear about the limitations of its evaluation.

Critical weaknesses

First, I am not convinced that the evaluation tests music decoding. Segments are split randomly without a temporal gap, so adjacent seconds of the same recording appear in both training and test, and the same excerpt heard by other subjects is in the training set. Identification could therefore rely on recording-specific context or memorized stimuli rather than musical content. Consistent with this, accuracy among 14 songs is nearly identical across encoders (0.68–0.69), and the gains appear mainly at the segment level.

Second, the theory is only loosely connected to the method, and some results raise concerns. The regression analysis concerns a fixed linear projection rather than learned tokenization, and none of the theoretical quantities is estimated in the experiments. The reconstruction claim rests on CLAP similarity, measured in the space the model is trained to predict, with a small margin over Postolache et al. (0.683 vs 0.672) and no listening study. In addition, Tables 2 and 8 report identical means for what are described as separate experiments, and the same reference run has different standard deviations in different tables.

Questions for authors

How do the results change under a song-held-out split, or at least a temporally blocked split with a gap? It would also be good to see how much leakage the current split allows, for instance with a baseline that predicts from temporally adjacent training segments. In addition, how are the 50 candidates chosen, and are they balanced by subject or by song? Some of the reported results are also unusual. For example, Tables 2 and 8 report identical means for what are described as separate experiments; why is this the case? Why is performance so low without pretraining? Variation across training seeds should also be reported. Finally, beyond the artifact removal in the preprocessing, residual muscle activity at the temporal channels should be ruled out before the attention maps are interpreted.

Reviewer_E3QCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors reconstruct the music from the EEG recorded during music presentation by applying channel-preserving representation alignment, separating electrode information retention from prediction regularization. The result indicates an improvement in correct music segmentation above prior methods. Fig. 3 is particularly interesting and compelling to me. The EEG aspect appears sound to me. The main issue i have is that the training and test data are not independent, coming from the same recordings.

Critical strengths

proposing a method to improve music generation by optimizing layer representation. Fig. 3 shows topography of attention distribution which is very helpful.

Critical weaknesses

the training and test data are not independent, coming from the same recordings.

Questions for authors

Have the authors tried another cross-validation approaches such as leaving out songs/participants?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.