acceptodds
Withdrawn from ICLR 2027

Hyperbolic Representation Learning for Multimodal-to-Unimodal Transfer in EEG-based Auditory Attention Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Auditory attention decoding (AAD) from electroencephalography (EEG) remains challenging across subjects, datasets, and recording systems. While auxiliary physiological signals can enrich EEG representations, their availability at inference time cannot be guaranteed. To solve this problem, we propose a codebook-guided pretraining framework that leverages paired EEG and auxiliary signals to learn transferable representations for EEG-only AAD. During pretraining, separate encoders map the two modalities into hyperbolic space, a hyperbolic residual codebook represents shared physiological structure with common codes and modality-specific information with residual codes. The shared center alignment objective further brings the two modalities' common representations into agreement. The framework then masks parts of the EEG representation and trains the EEG encoder to predict codebook targets learned from the paired multimodal data, thereby transferring the shared vocabulary to the EEG modality. For downstream AAD, the pretrained EEG encoder extracts features from EEG, which are passed to a classifier for attention prediction. Cross-subject and cross-dataset experiments on the AVGCAAD and EARAAD datasets demonstrate that the proposed model outperforms a range of state-of-the-art baselines, and ablation studies confirm that each key module contributes to the performance gains.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Auditory Attention Decoding, Electroencephalograph, Multimodal physiological signals, Hyperbolic representation learning, Vector quantization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_B3Y3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work examines EEG-based auditory attention decoding (AAD) under a practical setup: auxiliary signals (EOG or ear-EEG) are accessible during pretraining but unavailable during inference. The framework consists of three stages: (i) A joint tokenizer, (ii) An EEG-only encoder, (iii) The frozen encoder outputs. The authors evaluate leave-one-subject-out cross-subject generalization and cross-dataset transfer on two datasets: AVGCAAD (EEG+EOG, 12 participants) and EARAAD (scalp EEG + ear-EEG, 15 participants). Ablation experiments and hyperbolicity analysis are also included.

Critical strengths

The problem formulation is practically meaningful. Auxiliary recordings are frequently unavailable in real deployment, so leveraging them solely for training is a sensible research goal. Testing across two datasets with distinct auxiliary modalities (EOG versus ear-EEG) matches this motivation. The tokenizer supports variable channel counts, removing the requirement for identical channel layouts when performing cross-dataset transfer.

Critical weaknesses
  1. Reported accuracy values sit near chance level, and performance margins are far smaller than observed variance. The task is binary classification with a 50% chance baseline. Across-subject standard deviations on EARAAD range from 12 to 19 pp. With only 15 subjects, the standard error alone reaches roughly 3–3.5 pp. No statistical significance tests are provided. Therefore, the claim that the proposed method “outperforms a range of state-of-the-art baselines” lacks sufficient support.

  2. Critical ablation supporting the core claim is absent. The main contribution of this paper is multimodal-to-unimodal representation transfer. However, Figure 3 does not test the identical pipeline with a tokenizer trained only on EEG, i.e., without auxiliary signals. Without this comparison, any observed improvements (if genuine) may merely stem from LaBraM-style masked code prediction.

  3. The novelty claim is overstated, and relevant prior work is omitted. Training using a modality available only during training corresponds to the classic learning using privileged information / cross-modal distillation paradigm. A substantial body of literature already addresses missing-modality settings, including EEG studies using eye-tracking as privileged information. None of these references are cited, even though the first contribution is presented as a new formulation.

  4. The paper fails to address the well-known gaze confound in AVGCAAD, and standard AAD baselines are missing. The AV-GC-AAD dataset was specifically constructed to decouple gaze direction from auditory attention direction. Its original developers reported that spatial attention direction classifiers cannot achieve statistically significant above-chance performance on this dataset, while linear stimulus reconstruction works even in cross-subject settings. Two key implications arise: (a) All baselines adopted here are direction classifiers. (b) EOG serves as the auxiliary modality for the proposed method, and EOG directly encodes the gaze signal that the dataset was designed to eliminate.

Questions for authors
  1. Are performance improvements over EEGMOCE / LaBraM statistically significant via paired across-subject tests? How many individual subjects achieve accuracy significantly above chance?

  2. What performance can a linear stimulus-reconstruction baseline reach under identical windowing and evaluation protocol? What are split results for gaze-congruent and gaze-incongruent conditions on AVGCAAD?

  3. What happens when the tokenizer is pretrained using EEG alone (no auxiliary signals), keeping all other components unchanged? Is auxiliary data fed into the tokenizer when generating MCP target codes?

  4. In every LOSO fold, is the held-out subject excluded from both pretraining stages?

  5. How can a channel-permutation-invariant, mean-pooled encoder represent hemispheric lateralization? What do you get with channel embeddings added back?

Reviewer_aoFBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In their paper, the authors tackle auditory attention decoding (AAD) using EEG data under cross subject and cross dataset settings. The authors also argue that multi-modality during training can improve model performance but this is limited by the availability of the extra modalities during test time. As such, the paper aims at developing a modality agnostic model that trains using EEG and electrooculography (EOG) data for example but generalizes to the EEG only scenario. They do this in three stages. First, a tokenizer learns a shared hyperbolic codebook for EEG and the auxiliary signal, plus a separate residual codebook for each modality, with a loss that pulls the two modalities toward the same shared codes. They rely on hyperbolic space because they state that brain hierarchical cognitive states cannot be well represented in standard Euclidean space, and benefit from the distortion minimizing expanding distance metric of hyperbolic geometry. Second, an EEG-only encoder is pretrained to predict the tokenizer's shared codes for masked EEG patches using a Lorentz MLR classifier and a squared geodesic distance loss. Finally, a linear classifier is trained on top for the classification.

Critical strengths
  • The training only multi-modality scheme is a good problem to focus on, and it allows easier adoption of these models in actual settings.
  • Using channel-permutation-invariant encoding for the modalities makes the model more flexible.
Critical weaknesses
  • The paper seems to be missing important literature especially for hyperbolic and non-Euclidean models in EEG settings. Some prominent examples are MAtt [1], GyroAtt [2], LAtte [3]. Even EEG-MoCE which they cite is technically also hyperbolic. The division in table 1 makes this unclear, maybe hyperbolic geometry should be an extra column with true false since its not mutually exclusive with multimodal etc...

  • Some contributions in the paper have already been done in the literature and are not referenced. The Lorentz MLR for example was already established in [4] with a scaled version in [5]. The hyperbolic residual codebook is close to existing hyperbolic residual quantization [6] and VQ with codes tied to hyperbolic MLR hyperplanes [7]. Additionally, the stage-2 recipe (a tokenizer trained with an auxiliary signal, then masked prediction of its codes) follows BEiT v2 [8], and LaBraM already applies it to EEG. Using a modality only at training time has also been studied as privileged information [9], cross-modal distillation [10] and modality hallucination [11]. Apart from CM-SSL and Jiang et al. 2025, the paper does not discuss this work, so the formulation in contribution 1 is new only for AAD.

  • The cited Zenodo record (10.5281/zenodo.3618205) is the DTU dataset with 44 subjects and in-ear EEG for only 19 of them, and no around-ear EEG. The description in section 4.1 that has 15 subjects, 29 scalp + 3 EOG + 19 around-ear + 12 in-ear, six 10-minute trials, matches Geirnaert, Kappel and Kidmose (Sci. Rep. 2025; data at 10.5281/zenodo.16536441), where talkers are at ±60° and the attended side is fixed within each trial. Each subject then has about three left and three right trials, so the label is tied to trial identity and time, both of which are decodable. How labels and trial confounds were handled is not stated. Table 3 lists 34 auxiliary channels, but 19 + 12 = 31, how were the EOG channels included?

  • The results don't seem statistically significant. If the std here is across multiple seeds then I do not believe the results are significant. If they are across subject then multiple seeds still have to be run with a significance test. This is especially since model performance in EEG settings sometimes suffers from large performance differences.

  • The hyperbolic residual codebook seems to be the same as a Euclidean RVQ in the tangent space. This is because all the computations are done at the tangent space of the origin and the only non-Euclidean part is the nearest-code distance. Although that's also limited by the 2.0 norm clipping. It would be interesting to see a comparison to existing works on this like in [6,7].

There are some other minor issues throughout the paper like "They tried to explore" without mentioning who they is. Line 81 the datasets are mentioned without citation or any explanation. "comparision" in table 1 caption, "Eq. equation" "dounstream"... I think another look over the text for these errors would help

I am open to changing my score if these weaknesses are explained or addressed.

[1] Y.-T. Pan, J.-L. Chou, C.-S. Wei. "MAtt: A Manifold Attention Network for EEG Decoding."

[2] Wang, Hu, Song, Wu, Sebe, Chen. "GyroAtt: A Gyro Attention Framework for Matrix Manifolds"

[3] Bdeir, Burchert, Hanika, Schmidt-Thieme, Landwehr. "LAtte: Hyperbolic Lorentz Attention for Cross-Subject EEG Classification"

[4] Bdeir, Schwethelm, Landwehr. "Fully Hyperbolic Convolutional Neural Networks for Computer Vision."

[5] van der Klis, Chavez Torres, van Spengler, Ding, Hofmann, Mettes. "Fast and Geometrically Grounded Lorentz Neural Networks."

[6] Piekos, Kayal, Karatzoglou. "Hyperbolic Residual Quantization."

[7] Goswami, Mukuta, Harada. "HyperVQ: MLR-based Vector Quantization in Hyperbolic Space."

[8] Z. Peng, L. Dong, H. Bao, Q. Ye, F. Wei. "BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers."

[9] V. Vapnik, A. Vashist. "A New Learning Paradigm: Learning Using Privileged Information."

[10] S. Gupta, J. Hoffman, J. Malik. "Cross Modal Distillation for Supervision Transfer."

[11] J. Hoffman, S. Gupta, T. Darrell. "Learning with Side Information through Modality Hallucination."

Questions for authors
  1. Do you have per-subject accuracies, results over several seeds, and paired tests against the best baseline and against chance?
  2. Could you describe the LOSO setting? Does any pretraining see the held-out subject?
  3. HEEGnet introduced Domain-specific moment-then-distribution batch norm and Euclidean-Alignment. Was this attempted in this setting since it seemed to be a large contribution to performance in their work?
  4. Is there an ablation with an EEG-only tokenizer? What exactly does "w/o Hyperbolic" change?
Reviewer_boBiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem in auditory attention decoding where multimodal physiological signals are available during training but only EEG is available at inference. The authors propose a hyperbolic residual codebook that models multimodal information through a shared codebook and modality-specific residual codebooks, and transfers multimodal pretraining knowledge to the EEG encoder via shared-centroid alignment and masked codebook prediction. The paper conducts cross-subject and cross-dataset evaluations on AVGCAAD and EARAAD.

Critical strengths
  1. The research problem is practically meaningful. Auxiliary physiological signals may be unavailable at deployment, and leveraging them to improve EEG-only inference has clear application value.

  2. The overall framework design is relatively complete. Shared and residual quantization, cross-modal centroid alignment, and masked prediction form a coherent training pipeline. Aligning by temporal segments rather than channel-by-channel also accounts for the lack of channel correspondence across physiological signals.

  3. The experiments cover different auxiliary modalities and transfer settings. The authors consider both EOG and periauricular/in-ear EEG, and conduct component ablations, providing some support for assessing the framework's applicability.

Critical weaknesses
  1. The performance gains are small and statistical reliability has not been demonstrated. In Table 1, improvements over the best baseline in each setting are approximately 0.32–0.72 percentage points, while between-subject standard deviations are considerably larger. Large standard deviations alone do not negate the improvements, but subject-level paired tests, confidence intervals of the differences, and multi-seed results are currently lacking. The cross-dataset accuracies in Figure 2 are also only approximately 51%–53%, so the conclusion that "generalization ability is effectively improved" requires more caution, and it should be verified whether the results are reliably above chance level.

  2. Cross-dataset comparisons conflate differences in input channel counts. Appendix D states that some baselines use only the 27 EEG channels shared by both datasets, whereas the proposed method uses the full channel set. Therefore, current results cannot sufficiently distinguish whether performance gains arise from the representation learning method or from additional channel information. It is recommended to add comparisons where all methods use the same 27 channels, while retaining the full-channel experiments to demonstrate adaptability to different channel configurations.

The isolation between pretraining and test data is insufficiently clear. The appendix mentions leave-one-subject-out evaluation, but does not clearly state whether each fold relearns the codebook and performs masked pretraining using only training subjects, and how validation subjects are partitioned. Cross-dataset experiments also need to clarify whether the target data participates in any unsupervised training, normalization statistics estimation, or model selection. These details directly determine whether the experiments constitute strict inductive transfer or a setting that incorporates target-domain information.

  1. The actual contributions of auxiliary modalities versus shared/specific decomposition have not been sufficiently isolated. Current ablations cannot answer how much of the gain comes from auxiliary modalities versus from EEG-only codebook pretraining and masked learning. It is recommended to add an EEG-only pretraining control with the same architecture and training budget, as well as a control with shuffled EEG–auxiliary temporal pairings. Moreover, shared codebooks plus residual reconstruction do not automatically guarantee a semantic shared/specific decomposition; this interpretation still requires support from analyses such as codebook usage distributions, cross-modal encoding consistency, or cross-modal prediction.

  2. The necessity of hyperbolic geometry remains insufficiently justified. The lower relative hyperbolicity in Table 2 can serve as motivation, but is insufficient on its own to prove the existence of physiological hierarchical structure relevant to auditory attention decoding, nor does it directly imply that hyperbolic representations outperform Euclidean ones. The paper also needs to specify the distance metrics used for raw signals and EEGNet features, the sampling approach, and the multimodal combination method, and to explain the basis for the 0.30 threshold. Although Figure 3 includes a Euclidean ablation, it lacks exact values, error ranges, and clarification of whether training conditions are matched.

Questions for authors

In the leave-one-subject-out experiments, are codebook learning and masked pretraining completely excluded from test subjects? In the cross-dataset experiments, is the target dataset unused in all training stages?

After unifying to 27 EEG channels, does the proposed method still outperform the best baseline? Are the subject-level paired performance differences statistically significant?

Under the same training budget, what results are obtained with EEG-only pretraining and with shuffled cross-modal pairings, respectively? Can these controls support the core claim that "synchronized auxiliary signals provide effective transfer knowledge"?

Does the actual downstream evaluation employ linear probing with a frozen backbone or end-to-end fine-tuning? Do all baselines use a consistent evaluation protocol?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.