Multi-HiFormer: Multi-Scale Representation Learning with Hierarchical Transformer for Cross-Domain EEG Affective Analysis

Electroencephalography (EEG)-based affective analysis remains challenging due to the inherently multi-scale structure of neural responses, spanning temporal, spatial, and spectral dimensions. Existing approaches typically model these scales in isolation or depend on hand-crafted features, which limits their ability to learn representations that generalize across subjects, sessions, and tasks. We propose Multi-HiFormer, a hierarchical representation learning framework that models EEG along all three axes under a single unified principle: discriminative affective information is distributed across scales, and should be captured at every level of abstraction. Spatially, parallel convolutions with progressively larger receptive fields extract cortical patterns at complementary granularities. Spectrally, attention-based band calibration and cross-band interaction learn frequency compositions. Temporally, a hierarchical Transformer captures short, medium, and long-range dynamics via local convolution, masked attention, and global self-attention, grounding each temporal level in the multi-scale spatial and spectral features. In addition, we introduce physiology-informed channel-frequency calibration and dual-path representation fusion to enhance subject-specific neural feature learning. Beyond emotion recognition, Multi-HiFormer enables transfer learning to depression detection using the learned affective representations, demonstrating the potential of EEG emotion models for cross-task clinical applications. On a standard emotion EEG benchmark, Multi-HiFormer achieves F1, AUC, and accuracy of 0.8423, 0.8860, and 0.8442, respectively, outperforming the widely used EEGNet baseline (F1 = 0.6503, accuracy = 0.6525) by over 19 percentage points. These results suggest that multi-scale modeling provides a generalizable representation space for EEG-based affective computing.