- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
-
Key idea: The paper proposes EXIST, a cross-modal knowledge distillation framework for EEG-based emotion recognition in active emotional expression scenarios. This method addresses the modal discrepancies between audiovisual and EEG signals by introducing an audiovisual (AV)-guided EEG teacher model. This teacher model transmits emotion-related spatial and temporal information to the learner, who only receives EEG signals. Specifically, the proposed cortical-dependent distillation (CDD) supervises the distribution of attention between electrodes, while emotional dynamic distillation (EDD) guides temporal attention within the EEG window. Only the EEG signal is needed during inference.
-
Experiments on the EAV and PME4 datasets demonstrate that, under subject-independent evaluation conditions, EXIST improves upon existing EEG-based emotion recognition methods and traditional knowledge distillation methods. Furthermore, ablation experiments further validate the contributions of the proposed components.
Critical strengths
-
A clearly motivated problem: The paper addresses a practical challenge in EEG-based emotion recognition by leveraging audiovisual information during training while requiring only EEG signals at inference.
-
EEG-based native knowledge distillation: Instead of directly aligning heterogeneous audiovisual and EEG representations, the proposed framework distills spatial and temporal attention distributions within the EEG domain. This is a reasonable approach to mitigate cross-modal representation mismatch.
-
Complementary spatial and temporal supervision: CDD and EDD provide a unified framework that integrates audiovisual guidance into EEG spatial dependency modeling and temporal attention learning.
-
Comprehensive experimental comparison: The authors compared their proposed method with several baseline distillation methods based solely on EEG and cross-modal approaches. EXIST consistently outperforms the baseline methods on both datasets.
Critical weaknesses
-
Insufficient justification of cross-modal guidance: Although EXIST avoids direct feature alignment between audiovisual (AV) and electroencephalogram (EEG) representations, EEG teachers still rely on cross-modal affinity scores calculated through learned linear projections. It remains unclear whether these scores capture meaningful emotional correspondences between the two modalities or are merely task-specific statistical correlations. Therefore, claiming that modal differences have been “resolved by design” may be an overstatement.
-
Insufficient justification of the PCC-based connectivity prior: The proposed PCC prior is obtained by calculating the correlation between five-dimensional DE band features, which may not adequately capture the functional connectivity between EEG electrodes. Furthermore, applying the ReLU activation function removes all negative correlations, potentially losing some valuable inter-electrode relationships. The authors should provide stronger justification for these design choices or compare them with other connectivity metrics.
-
Novelty and design justification: The proposed CDD and EDD methods extend existing attention mechanisms by introducing AV-guided bias terms into spatial and temporal attention scores, followed by KL-based attention distillation. While their integration into an EEG-native distillation framework is interesting, the rationale behind several design choices could be further justified, as alternative configurations may offer comparable or better performance. For instance, why is distillation performed on the cortical dependency matrix S rather than the final GCN adjacency matrix A? Additionally, the row-wise injection baseline in Table A.4 requires clarification: if the bias is added uniformly to each row before row-wise softmax, it mathematically cancels out, raising concerns about the validity of this comparison.
-
Limited experimental generalizability and statistical validation:
-
The relatively small number of participants in both datasets (42 in the EAV dataset and only 11 in the PME4 dataset) limits the diversity of the evaluated population.
-
This evaluation relies on only three random, subject-independent sample splits, each using three training seeds, which may be insufficient to characterize inter-subject differences, especially for PME4. Furthermore, the lack of statistical significance tests and subject-level performance analysis limits the assessment of model robustness. Other evaluation methods, such as leave-one-subject cross-validation (LOSO), would help enhance the reliability of the conclusions.
-
The experiments on PME4 only utilize audio and EEG data, meaning the complete audiovisual guidance framework was effectively evaluated on only one dataset (EAV). It would be valuable to explain why other multimodal emotion datasets, such as PEGConv and MAHNOB-HCI, were not considered for additional validation.
-
Insufficient interpretability validation: The authors interpret the learned spatial attention distribution as cortical dependence related to emotion and the temporal attention weights as moments when emotion is salient. However, the qualitative visualizations in Figure 5 are insufficient to demonstrate that these attention patterns reflect meaningful neural activity or truly contribute to emotion recognition.
-
Incomplete knowledge transfer from teacher to student: The proposed knowledge extraction framework exhibits a significant performance gap between EAV-guided EEG teachers and students using only EEG. For example, in the EAV test, teachers achieved an accuracy rate of 50.33%, while students achieved only 41.49%. Further analysis of factors limiting knowledge transfer and whether alternative knowledge extraction strategies can narrow this gap will contribute to a more comprehensive evaluation of the proposed framework.
-
Limited temporal granularity of audiovisual guidance: EDD relies on globally shared audiovisual context to guide attention across EEG time windows, which may result in the loss of fine temporal information. Whether this design can effectively capture emotionally meaningful temporal dynamics compared to using time-aligned window-level audiovisual features remains unclear.
-
Unclear EEG preprocessing details: The paper extracts DE features from five frequency bands, while PME4 EEG signals are band-pass filtered at 0.1–30 Hz. Since the conventional gamma band extends above 30 Hz, the authors should clarify the frequency-band definitions and how the gamma-band features are extracted for PME4.
Questions for authors
-
How do the authors justify that these scores capture meaningful emotional correspondence rather than task-specific statistical correlations? In what sense is the modality gap “resolved by design”?
-
What is the motivation for calculating the PCC prior using only five DE band features and removing all negative correlations? Did the authors consider other connectivity metrics or retaining negative correlations to evaluate whether these design choices are optimal?
-
Why is the distillation operation performed on the cortical dependency matrix (S) instead of the final GCN adjacency matrix (A)? Furthermore, could the authors explain the specific implementation of the row-by-row injection baseline in Table A.4? If the bias term is added uniformly to each row before the row-by-row softmax, then mathematically, the bias term would cancel out.
-
Could the authors provide segmentation or subject-level results, along with statistical significance analysis, to support the reported improvements? Furthermore, why was the complete audiovisual guidance framework evaluated only on the EAV dataset, rather than other multimodal dialogue datasets such as PEGConv and MAHNOB-HCI?
-
Why did the authors use a globally shared AV context for EDD analysis instead of time-aligned, window-level AV features?
-
Can the authors provide more evidence that the learned spatial and temporal attention patterns do indeed help with emotion recognition, for example, through experiments involving electrode or time window perturbation?
-
Given the performance gap between EAV-guided EEG teachers (50.33%) and students using only EEG (41.49%) in the EAV test, what are the main factors limiting knowledge transfer?
-
How are the five DE frequency-band features defined for PME4, given that its EEG signals are filtered at 0.1–30 Hz? In particular, how are gamma-band features extracted under this preprocessing configuration?
The authors have addressed most of the concerns quite well, even though I still have one minor concern regarding the statistical significance analysis. I'm adjusting my rating to weak accept.
10 comments
W3 & Q3: Structural Distillation and Injection Design.
Response: We thank the reviewer for these insightful comments.
(1) Why distill S rather than A?
We choose to distill because it directly represents the electrode dependencies learned from AV guidance, whereas further incorporates the static PCC prior and self-connections. The two targets optimize related but conceptually different objectives: distilling isolates the AV-guided component, whereas distilling matches the complete propagation graph.
We also evaluated -level distillation for comparison. Table R3 shows that both targets achieve comparable performance. We conducted a subject-level paired significance test (Table R4). The experiment does not provide evidence that either target is empirically superior; instead, it shows that CDD is robust to this design choice.
We retain as the default because it more precisely isolates the AV-guided knowledge that CDD is designed to transfer.
Table R3: Comparison of matrix S and matrix A as CDD Distillation Targets.
| CDD Distillation Target | Accuracy (%) | Weighted F1 (%) |
|---|---|---|
| Matrix S | 41.49 ± 2.33 | 40.04 ± 1.42 |
| Matrix A | 41.50 ± 1.80 | 40.34 ± 1.66 |
Table R4: Subject-Level Paired Comparison of S- and A-Matrix Distillation on EAV.
| Metric | 95% CI (pp) | p-value | A Wins / Ties / Losses |
|---|---|---|---|
| Acc | [-0.92, +0.93] | 0.9837 | 13 / 1 / 13 |
| F1 | [-0.61, +1.24] | 0.4876 | 17 / 0 / 10 |
(2) Clarification of the Row-wise Injection Method
We would like to clarify the mathematical formulations and mechanisms of the row-wise injection strategy.
Row-wise Injection. Row-wise injection uses the AV-guided resonance score to modulate the attention that electrode assigns to other electrodes. Specifically,
Here, is added to all off-diagonal attention scores in the -th row. This shifts more attention from the electrode itself to other electrodes when its AV resonance is high. We will add these details to the revised manuscript.
W4 & Q4: Subject-level results and statistical validation. & Dataset selection.
Response: We thank the reviewer for raising this important question.
(1) Subject-level results and statistical validation: We conducted a paired subject-level analysis comparing the performance between EXIST and the strongest baseline(DGCNN on EAV, EEG-SCMM on PME4). The results in Table R5 and R6 demonstrate that our improvements are statistically significant.
As shown in Table R5, across 27 subject–split evaluations, EXIST outperforms DGCNN in 19 cases for accuracy and 18 cases for F1 on the EAV dataset. Paired t-tests confirm the gains are statistically significant (Acc: p = 0.0366; F1: p = 0.0463). This provides statistical evidence that EXIST yields improvements across the strongest baseline method on EAV.
As shown in Table R6, across 6 subject–split evaluations, EXIST outperforms EEG-SCMM in all cases for both Acc and F1 on PME4 dataset. Paired t-tests confirm the gains are statistically significant (Acc: p = 0.0029; F1: p = 0.0343). This provides statistical evidence that EXIST yields improvements across the strongest baseline method on PME4.
These results provide statistical evidence for the effectiveness of EXIST.
(2) Why wasn't the video modality used for the PME4 dataset, and why were the PEGConv and MAHNOB-HCI datasets not chosen for the evaluation:
Although PME4 includes facial video recordings, the dataset only provides pre-extracted visual features from VGG16 for 3 subjects, as not all participants consented to sharing their video data. Therefore, we use audio-only guidance on PME4.
MAHNOB-HCI follows an emotion-elicitation paradigm: participants watch affective videos while their facial responses and EEG are recorded. This is different from our setting, which focuses on actively expressed emotions in more interactive and clinically relevant scenarios. In this elicitation setting, it remains unclear whether facial expressions provide stronger emotional cues than EEG responses. In addition, the audio in MAHNOB-HCI mainly comes from the stimulus videos rather than the participants themselves, so the full audiovisual guidance used in EXIST cannot be directly applied. So we didn't choose it as our evaluation dataset.
PEGConv is very relevant to our setting because it records synchronized EEG, audio, and video during conversations. However, access requires a signed agreement from the data provider, which they don't release. We also contacted the authors but did not receive a response. We therefore could not include PEGConv in the current experiments. If access becomes available, we will evaluate EXIST on this dataset in future work.
To Reviewer rJdE
We thank the reviewer for the detailed and constructive feedback. We address each point below.
W1 & Q1: whether these scores capture meaningful emotional correspondences between the two modalities or are merely task-specific statistical correlations? In what sense is the modality gap “resolved by design”?
Response: We thank the reviewer for this concern.
(1) Meaningfulness of cross-modal resonance scores: We conducted an AV shuffling experiment, which provides evidence that the learned resonance scores capture meaningful emotional correspondence between EEG and AV modalities, rather than merely task-specific statistical correlations. We replaced each EEG sample's paired AV input with mismatched AV inputs during training. Specifically, we consider five settings: (a) paired AV guidance, (b) without AV guidance, (c) within-class shuffled AV guidance, (d) different-class shuffled AV guidance, and (e) fully shuffled AV guidance.
The results in Table R1 provide several observations:
(a) vs. (b): Paired AV guidance improves accuracy by 2.77%, indicating that the paired AV-guidance provides useful information for emotion recognition.
(a) vs. (c): Shuffling AV guidance within the same emotion class reduces accuracy by 2.06%, indicating the effectiveness of cross-modal guidance depends on correct sample-level paired EEG-AV correspondence beyond shared emotion labels.
(c) vs. (d): Within-class shuffled guidance outperforms different-class guidance by 1.20%, indicating that emotion-class consistency between AV and EEG also contributes to effective cross-modal guidance.
(c) vs. (e): Within-class shuffled guidance outperforms fully shuffled guidance by 1.08%, indicating that emotion-class-consistent AV guidance is more useful than the guidance from arbitrary AV information, further supporting the usefulness of emotion-class consistency.
(b) vs. (e): Fully shuffled AV guidance performs worse than no AV guidance, suggesting that the misleading guidance may harm the performance of EEG-student.
These results provide evidence that the learned affinity scores capture cross-modal affective correspondence beyond class-level and nuisance-driven statistical correlations.
Table R1: Effect of AV Guidance and Cross-Modal Mismatching.
(2) Clarification about "modality gap is resolved by design”: We apologize for the inaccurate use of "resolved" in the original manuscript. Our intention was to emphasize that EXIST introduces an AV-guided EEG teacher to enable EEG-native knowledge distillation, avoiding direct alignment between heterogeneous AV and EEG representations and thereby mitigating the modality gap. We will replace "resolved" with "mitigated" in the revised manuscript.
W2 & Q2: What is the motivation for calculating the PCC prior using five DE features and removing all negative correlations? Did the authors consider other connectivity metrics or retaining negative correlations to evaluate whether these designs are optimal?
Response:
Motivation for DE-based PCC and ReLU: In EXIST, the PCC-based prior is used to construct statistical correlations between EEG electrodes, rather than to directly estimate physiological brain connectivity. Specifically, DE-based PCC measures the similarity of spectral feature patterns across electrodes, providing an initial graph structure to guide feature aggregation in GCN. We apply ReLU to obtain a non-negative adjacency matrix compatible with standard degree-normalized graph convolution.
Our choice of DE PCC (ReLU) was based on empirical comparisons of several spatial prior construction methods, including PCC from raw EEG and DE features with absolute-value or ReLU transformations, as well as PLV. As shown in Table R2, DE PCC (ReLU) achieved the best overall performance among these alternatives, which supports our choice. All evaluated priors improve performance over the model without a spatial prior, supporting the effectiveness of introducing an initial static prior to model inter-electrode relationships.
Table R2: Comparison for PCC construction.