acceptodds
Under review as a conference paper at ICLR 2027

Multi-HiFormer: Multi-Scale Representation Learning with Hierarchical Transformer for Cross-Domain EEG Affective Analysis

Abstract

Electroencephalography (EEG)-based affective analysis remains challenging due to the inherently multi-scale structure of neural responses, spanning temporal, spatial, and spectral dimensions. Existing approaches typically model these scales in isolation or depend on hand-crafted features, which limits their ability to learn representations that generalize across subjects, sessions, and tasks. We propose Multi-HiFormer, a hierarchical representation learning framework that models EEG along all three axes under a single unified principle: discriminative affective information is distributed across scales, and should be captured at every level of abstraction. Spatially, parallel convolutions with progressively larger receptive fields extract cortical patterns at complementary granularities. Spectrally, attention-based band calibration and cross-band interaction learn frequency compositions. Temporally, a hierarchical Transformer captures short, medium, and long-range dynamics via local convolution, masked attention, and global self-attention, grounding each temporal level in the multi-scale spatial and spectral features. In addition, we introduce physiology-informed channel-frequency calibration and dual-path representation fusion to enhance subject-specific neural feature learning. Beyond emotion recognition, Multi-HiFormer enables transfer learning to depression detection using the learned affective representations, demonstrating the potential of EEG emotion models for cross-task clinical applications. On a standard emotion EEG benchmark, Multi-HiFormer achieves F1, AUC, and accuracy of 0.8423, 0.8860, and 0.8442, respectively, outperforming the widely used EEGNet baseline (F1 = 0.6503, accuracy = 0.6525) by over 19 percentage points. These results suggest that multi-scale modeling provides a generalizable representation space for EEG-based affective computing.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG, Affective analysis, Hierarchical Transformer, Multi-scale modeling, Transfer learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DPRuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper stacks together several known components for EEG emotion recognition: learnable channel/band weights initialized from depression biomarkers, five parallel convolutions over channels (kernel 3-11), cross-attention over five frequency bands, a BiGRU plus a three-branch temporal transformer (local conv / windowed attention / global attention) with a gate, and a bypass path carrying raw band statistics. On BME2026 it reports ~84% accuracy vs ~74% for the best baseline, 84.65% on SEED, and a transfer experiment to depression screening, with a large amount of ablation, synergy and robustness analysis.

Critical strengths
  1. The evaluation is broad: HC/DEP subgroup accuracy for every baseline, an external dataset, a transfer experiment. More effort than typical for this area.
  2. Reporting subgroup accuracy alongside overall accuracy is a good habit for anything close to clinical use.
Critical weaknesses
  1. The method is a stack of standard parts, and the one thing claimed as novel is never tested. Eqs 1-5 are textbook items (a residual linear layer, vanilla softmax attention, standard attention pooling); multi-kernel conv, SE blocks, BiGRU, gated fusion are all off the shelf. The physiology-informed initialization is listed as a main contribution but there is no random-init vs biomarker-init ablation anywhere, and the paper's own Fig 4 puts the largest group differences in delta/alpha, not the beta/gamma prior it initializes from. Some details don't match the described setup either: 62 channels recorded, a "30-channel subset" claimed, yet the bypass input in Sec 3.5 is 310 = 62x5, and the channel attention analyses cover 20 channels.
  2. The paper's own tables disagree with each other. Removing DualPathFusion costs 7.3 pp in Table 3, but 22.8 pp under clean conditions in Table 6 / Sec 4.5.3. In Table 5 single-module removals are mostly near zero or negative (removing a module helps?), which contradicts Table 3 (-4.2 to -7.3 pp) and the claim that every ablation costs accuracy; the same module even gets three different single-removal numbers inside Table 5. Sec 4.8 says the full model trains in 102 min/seed while the smaller ablation arms take 6-9 min each - I don't see how smaller models train ~13x faster than the full one. Sec 4.5.1 says gradients are non-zero "from early epochs onward", but Table 4 shows 0.0 for all modules at epoch 1 (the appendix itself says "from epoch 20 onward"). The value 0.755 appears as three supposedly different quantities: DEP subgroup accuracy in Table 1, the depression transfer accuracy in Sec 4.6 (with the same +12.1 pp margin as Sec 4.2), and the per-subject mean in Fig 3C, which should sit somewhere near the 0.844 overall accuracy instead. Sec 4.5.3 claims the full model stays above five of six variants under 30% channel dropout, but computing from Table 6 two variants are above it. Counting is off in places too ("six ablated variants" vs five arms in Table 3; "seven variants" for a six-row Table 7).
Questions for authors
  1. Please remake the figures with legible text, as the current text is difficult to read. The writing and figure-drawing skills require improvement.
  2. Why do the smaller ablation models train ~13x faster than the full model (6-9 min vs 102 min per seed)?
  3. How many channels did the model actually take as input (62, 30, or the 62x5 raw features), and why do Figs 5-6 show 20?
  4. Is 0.755 in Table 1, Sec 4.6 and Fig 3C really three independent measurements?
  5. Please add a random-init vs biomarker-init comparison for the calibration masks; without it contribution 3 is unsupported.
  6. Which number is right for removing DualPathFusion, -7.3 or -22.8? And were Table 5's single-removal numbers produced under the Sec 4.1 protocol at all?
Reviewer_oz2cOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Multi-HiFormer, a multi-scale framework for cross-subject EEG emotion recognition. The model integrates a Spatial Scale Pyramid (SSP) with parallel convolutions, a Spectral Token Mixer (STM) for cross-band interactions, and a Hierarchical Temporal Transformer (HTT) for capturing short, medium, and long-range temporal dependencies. Additionally, physiology-informed calibration and dual-path feature fusion are introduced to enhance EEG representation learning.

The proposed method is evaluated on the BME2026 and SEED datasets, achieving classification accuracies of 84.42% and 84.65%, respectively. The authors compare the model against several conventional machine learning and deep learning baselines and conduct ablation studies, robustness evaluations, and representation analyses. The paper also explores the potential transfer of emotion-related representations to depression detection.

Critical strengths
  1. Practical relevance of the research problem. The paper addresses cross-subject EEG emotion recognition, an important problem in affective computing. The authors also examine the generalizability of the method to emotion recognition in depressive subjects.

  2. Relatively comprehensive ablation studies. Beyond the main classification results, the authors conduct component ablation studies, robustness evaluations, and representation analyses to investigate the proposed architecture. These experiments provide additional insights into the contributions and behavior of different components.

Critical weaknesses
  1. Limited Methodological Novelty

The methodological novelty of this work is relatively limited. The proposed framework primarily integrates existing techniques, including multi-scale convolutions, cross-band attention, BiGRU, hierarchical Transformers, and feature fusion. Several closely related studies have already explored similar ideas:

  • AMDET (IEEE TAC, 2023) investigates joint spectral-spatial-temporal EEG representation learning using multidimensional attention mechanisms.
  • R2G-STLT (Neural Networks, 2024) proposes hierarchical spatial-temporal Transformer modeling from regional to global brain structures.
  • EmT (IEEE TNNLS, 2025) combines multi-view pyramid graph convolution with temporal Transformers for cross-subject EEG emotion recognition.

These studies share substantial similarities with Multi-HiFormer in their core modeling ideas and certain architectural designs. However, the paper does not adequately distinguish its contributions from these existing approaches or clearly demonstrate a fundamentally new modeling mechanism. Consequently, the proposed method appears to be an incremental integration of established components rather than a substantial methodological advance.

  1. Insufficient Experimental Comparisons and Concerns About Result Reliability

The experimental evaluation raises several concerns regarding the reliability of the reported performance improvements.

First, the baseline comparisons are insufficient, particularly on BME2026, where the authors mainly compare against conventional machine learning methods and relatively outdated deep learning architectures. More competitive and recent EEG emotion recognition methods are missing from the evaluation.

Second, there are inconsistencies in the ablation studies. The reported effects of removing individual modules differ substantially between Table 3 and Appendix Table 5.

Questions for authors
  1. Could the authors compare with state-of-the-art methods on cross-subject emotion recognition?

  2. The ablation results are inconsistent. Single-module removal effects differ substantially between Table 3 and Appendix Table 5. The sign convention in Table 5 is also unclear: several double-removal results appear to exceed the full model, contrary to the accompanying text. Please clarify these discrepancies before drawing conclusions about module necessity and synergy.

  3. Figure 1 is difficult to read because the text is too small. Some modules, such as the Hierarchical Temporal Transformer, appear twice in the figure. Please simplify the figure and highlight the most critical elements.

  4. Section 4.6 reports a depression detection accuracy of approximately 75.5%, which closely matches the depressed-subgroup emotion recognition accuracy in Table 1. Could the authors clarify the evaluation protocol and provide independent subject-level depression classification results to substantiate the claimed cross-task transfer capability? Could the authors compare with other state-of-the-art depression detection methods, and illustrate the necessity of transferring an emotion recognition model to depression detection?

  5. Section 4.1 states that the evaluated model uses a 30-channel EEG subset, whereas Section 3.5 describes a raw spectral feature projection with an input dimension of 310, corresponding to 62 channels across five frequency bands. Could the authors clarify this discrepancy and specify which channel configuration is used in each processing pathway?

Reviewer_REhvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Multi-HiFormer, a hierarchical framework for EEG-based affective analysis, with a primary focus on cross-subject emotion recognition. The architecture combines physiology-informed channel and frequency-band calibration, parallel spatial convolutions with different kernel sizes, cross-band attention, and BiGRU encoder followed by HTT. The HTT uses gated fusion to combine short, medium, and long-range temporal branches. Physiological priors are incorporated as learnable calibration masks, while a dual-path fusion mechanism combines the learned hierarchical representation with input-level spectral statistics. The authors evaluate the model on BME2026 and SEED and report improvements over the selected baselines. They also present preliminary evidence of transfer from emotion recognition to depression detection.

Critical strengths

1.This work presents an EEG-based affective analysis framework. It accounts for substantial inter-subject variability and for complementary information across channels, frequency bands, and spatial dimensions; the proposed multiscale spatial processing framework is a sensible way to address this problem. Keeping a complementary statistical representation alongside the learned hierarchy is also a good design choice. 2.The work achieves solid gains on BME2026 and also performs well on the SEED dataset, with clear improvements on several metrics.

Critical weaknesses

1.The SEED experiments use different evaluation protocols, so they cannot fully support the authors' claims about generalization based on emotion assessment. In addition, the claim of transfer to depression detection lacks a separately identifiable and sufficiently specified experiment. The experiments feel incomplete, and the sample sizes in the datasets are also relatively small. 2.There is a lack of comparison and contrast with more recent work, for example MMASE-DG and EmT.

Questions for authors

1.Could larger test datasets be used to support your conclusions? 2.Are there any recent works that could be compared against for performance? 3.Could you specify the number of training samples and the size of the validation set?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.