acceptodds
Under review as a conference paper at ICLR 2027

EXIST: Explicit-Guided Spatiotemporal Supervision for EEG Emotion Recognition

Abstract

Emotion recognition in active emotional expression scenarios, such as clinical interviews, relies on audio and video, which provide rich affective cues, but cannot be collected from privacy-sensitive patients. EEG offers a privacy-preserving alternative, yet EEG-based emotion recognition falls behind audiovisual systems due to low signal-to-noise ratio and strong inter-subject variability. We propose explicit-to-implicit supervision, a distillation paradigm that leverages explicit modalities (video and audio) to supervise the implicit modality (EEG) during training and discards them at inference time. Current distillation methods cause negative transfer in this context because they align features or logits across incompatible spaces, and neglect the spatiotemporal structure of the EEG. Therefore, we propose EXIST (EXplicit-guided Implicit SpatioTemporal supervision for EEG), which performs EEG-native reasoning using audiovisual signals as structural supervision within the EEG manifold, enabling EEG to-EEG transfer while avoiding modality conflict. In particular, we design two components: Cortical Dependency Distillation (CDD) and Emotional Dynamics Distillation (EDD). CDD transfers emotion-relevant cortical dependency priors inferred from audiovisual context into EEG-native spatial reasoning, whereas EDD focuses on learning emotionally salient temporal dynamics rather than enforcing direct imitation across modalities. Experiments on EAV and PME4 demonstrate that EXIST achieves state-of-the-art EEG-only emotion recognition under a subject-independent protocol, outperforming both prior EEG-only and existing cross-modal distillation methods.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG, Emotion Recognition, Spatiotemporal Supervision

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rJdEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Key idea: The paper proposes EXIST, a cross-modal knowledge distillation framework for EEG-based emotion recognition in active emotional expression scenarios. This method addresses the modal discrepancies between audiovisual and EEG signals by introducing an audiovisual (AV)-guided EEG teacher model. This teacher model transmits emotion-related spatial and temporal information to the learner, who only receives EEG signals. Specifically, the proposed cortical-dependent distillation (CDD) supervises the distribution of attention between electrodes, while emotional dynamic distillation (EDD) guides temporal attention within the EEG window. Only the EEG signal is needed during inference.

  • Experiments on the EAV and PME4 datasets demonstrate that, under subject-independent evaluation conditions, EXIST improves upon existing EEG-based emotion recognition methods and traditional knowledge distillation methods. Furthermore, ablation experiments further validate the contributions of the proposed components.

Critical strengths
  1. A clearly motivated problem: The paper addresses a practical challenge in EEG-based emotion recognition by leveraging audiovisual information during training while requiring only EEG signals at inference.

  2. EEG-based native knowledge distillation: Instead of directly aligning heterogeneous audiovisual and EEG representations, the proposed framework distills spatial and temporal attention distributions within the EEG domain. This is a reasonable approach to mitigate cross-modal representation mismatch.

  3. Complementary spatial and temporal supervision: CDD and EDD provide a unified framework that integrates audiovisual guidance into EEG spatial dependency modeling and temporal attention learning.

  4. Comprehensive experimental comparison: The authors compared their proposed method with several baseline distillation methods based solely on EEG and cross-modal approaches. EXIST consistently outperforms the baseline methods on both datasets.

Critical weaknesses
  1. Insufficient justification of cross-modal guidance: Although EXIST avoids direct feature alignment between audiovisual (AV) and electroencephalogram (EEG) representations, EEG teachers still rely on cross-modal affinity scores calculated through learned linear projections. It remains unclear whether these scores capture meaningful emotional correspondences between the two modalities or are merely task-specific statistical correlations. Therefore, claiming that modal differences have been “resolved by design” may be an overstatement.

  2. Insufficient justification of the PCC-based connectivity prior: The proposed PCC prior is obtained by calculating the correlation between five-dimensional DE band features, which may not adequately capture the functional connectivity between EEG electrodes. Furthermore, applying the ReLU activation function removes all negative correlations, potentially losing some valuable inter-electrode relationships. The authors should provide stronger justification for these design choices or compare them with other connectivity metrics.

  3. Novelty and design justification: The proposed CDD and EDD methods extend existing attention mechanisms by introducing AV-guided bias terms into spatial and temporal attention scores, followed by KL-based attention distillation. While their integration into an EEG-native distillation framework is interesting, the rationale behind several design choices could be further justified, as alternative configurations may offer comparable or better performance. For instance, why is distillation performed on the cortical dependency matrix S rather than the final GCN adjacency matrix A? Additionally, the row-wise injection baseline in Table A.4 requires clarification: if the bias is added uniformly to each row before row-wise softmax, it mathematically cancels out, raising concerns about the validity of this comparison.

  4. Limited experimental generalizability and statistical validation:

  • The relatively small number of participants in both datasets (42 in the EAV dataset and only 11 in the PME4 dataset) limits the diversity of the evaluated population.

  • This evaluation relies on only three random, subject-independent sample splits, each using three training seeds, which may be insufficient to characterize inter-subject differences, especially for PME4. Furthermore, the lack of statistical significance tests and subject-level performance analysis limits the assessment of model robustness. Other evaluation methods, such as leave-one-subject cross-validation (LOSO), would help enhance the reliability of the conclusions.

  • The experiments on PME4 only utilize audio and EEG data, meaning the complete audiovisual guidance framework was effectively evaluated on only one dataset (EAV). It would be valuable to explain why other multimodal emotion datasets, such as PEGConv and MAHNOB-HCI, were not considered for additional validation.

  1. Insufficient interpretability validation: The authors interpret the learned spatial attention distribution as cortical dependence related to emotion and the temporal attention weights as moments when emotion is salient. However, the qualitative visualizations in Figure 5 are insufficient to demonstrate that these attention patterns reflect meaningful neural activity or truly contribute to emotion recognition.

  2. Incomplete knowledge transfer from teacher to student: The proposed knowledge extraction framework exhibits a significant performance gap between EAV-guided EEG teachers and students using only EEG. For example, in the EAV test, teachers achieved an accuracy rate of 50.33%, while students achieved only 41.49%. Further analysis of factors limiting knowledge transfer and whether alternative knowledge extraction strategies can narrow this gap will contribute to a more comprehensive evaluation of the proposed framework.

  3. Limited temporal granularity of audiovisual guidance: EDD relies on globally shared audiovisual context to guide attention across EEG time windows, which may result in the loss of fine temporal information. Whether this design can effectively capture emotionally meaningful temporal dynamics compared to using time-aligned window-level audiovisual features remains unclear.

  4. Unclear EEG preprocessing details: The paper extracts DE features from five frequency bands, while PME4 EEG signals are band-pass filtered at 0.1–30 Hz. Since the conventional gamma band extends above 30 Hz, the authors should clarify the frequency-band definitions and how the gamma-band features are extracted for PME4.

Questions for authors
  1. How do the authors justify that these scores capture meaningful emotional correspondence rather than task-specific statistical correlations? In what sense is the modality gap “resolved by design”?

  2. What is the motivation for calculating the PCC prior using only five DE band features and removing all negative correlations? Did the authors consider other connectivity metrics or retaining negative correlations to evaluate whether these design choices are optimal?

  3. Why is the distillation operation performed on the cortical dependency matrix (S) instead of the final GCN adjacency matrix (A)? Furthermore, could the authors explain the specific implementation of the row-by-row injection baseline in Table A.4? If the bias term is added uniformly to each row before the row-by-row softmax, then mathematically, the bias term would cancel out.

  4. Could the authors provide segmentation or subject-level results, along with statistical significance analysis, to support the reported improvements? Furthermore, why was the complete audiovisual guidance framework evaluated only on the EAV dataset, rather than other multimodal dialogue datasets such as PEGConv and MAHNOB-HCI?

  5. Why did the authors use a globally shared AV context for EDD analysis instead of time-aligned, window-level AV features?

  6. Can the authors provide more evidence that the learned spatial and temporal attention patterns do indeed help with emotion recognition, for example, through experiments involving electrode or time window perturbation?

  7. Given the performance gap between EAV-guided EEG teachers (50.33%) and students using only EEG (41.49%) in the EAV test, what are the main factors limiting knowledge transfer?

  8. How are the five DE frequency-band features defined for PME4, given that its EEG signals are filtered at 0.1–30 Hz? In particular, how are gamma-band features extracted under this preprocessing configuration?

The authors have addressed most of the concerns quite well, even though I still have one minor concern regarding the statistical significance analysis. I'm adjusting my rating to weak accept.

10 comments

To Reviewer rJdE

We thank the reviewer for the detailed and constructive feedback. We address each point below.

W1 & Q1: whether these scores capture meaningful emotional correspondences between the two modalities or are merely task-specific statistical correlations? In what sense is the modality gap “resolved by design”?

Response: We thank the reviewer for this concern.
(1) Meaningfulness of cross-modal resonance scores: We conducted an AV shuffling experiment, which provides evidence that the learned resonance scores capture meaningful emotional correspondence between EEG and AV modalities, rather than merely task-specific statistical correlations. We replaced each EEG sample's paired AV input with mismatched AV inputs during training. Specifically, we consider five settings: (a) paired AV guidance, (b) without AV guidance, (c) within-class shuffled AV guidance, (d) different-class shuffled AV guidance, and (e) fully shuffled AV guidance.

The results in Table R1 provide several observations:

(a) vs. (b): Paired AV guidance improves accuracy by 2.77%, indicating that the paired AV-guidance provides useful information for emotion recognition.

(a) vs. (c): Shuffling AV guidance within the same emotion class reduces accuracy by 2.06%, indicating the effectiveness of cross-modal guidance depends on correct sample-level paired EEG-AV correspondence beyond shared emotion labels.

(c) vs. (d): Within-class shuffled guidance outperforms different-class guidance by 1.20%, indicating that emotion-class consistency between AV and EEG also contributes to effective cross-modal guidance.

(c) vs. (e): Within-class shuffled guidance outperforms fully shuffled guidance by 1.08%, indicating that emotion-class-consistent AV guidance is more useful than the guidance from arbitrary AV information, further supporting the usefulness of emotion-class consistency.

(b) vs. (e): Fully shuffled AV guidance performs worse than no AV guidance, suggesting that the misleading guidance may harm the performance of EEG-student.

These results provide evidence that the learned affinity scores capture cross-modal affective correspondence beyond class-level and nuisance-driven statistical correlations.

Table R1: Effect of AV Guidance and Cross-Modal Mismatching.

No.AV GuidanceAcc. (%)Weighted F1 (%)
(a)Paired AV guidance41.49 ± 2.3340.04 ± 1.42
(b)Without AV guidance38.72 ± 2.5838.04 ± 2.55
(c)Within-class shuffled AV guidance39.43 ± 2.4538.32 ± 2.49
(d)Different-class shuffled AV guidance38.23 ± 2.5237.42 ± 1.97
(e)Fully shuffled AV guidance38.35 ± 2.6537.81 ± 2.33

(2) Clarification about "modality gap is resolved by design”: We apologize for the inaccurate use of "resolved" in the original manuscript. Our intention was to emphasize that EXIST introduces an AV-guided EEG teacher to enable EEG-native knowledge distillation, avoiding direct alignment between heterogeneous AV and EEG representations and thereby mitigating the modality gap. We will replace "resolved" with "mitigated" in the revised manuscript.

W2 & Q2: What is the motivation for calculating the PCC prior using five DE features and removing all negative correlations? Did the authors consider other connectivity metrics or retaining negative correlations to evaluate whether these designs are optimal?

Response:
Motivation for DE-based PCC and ReLU: In EXIST, the PCC-based prior is used to construct statistical correlations between EEG electrodes, rather than to directly estimate physiological brain connectivity. Specifically, DE-based PCC measures the similarity of spectral feature patterns across electrodes, providing an initial graph structure to guide feature aggregation in GCN. We apply ReLU to obtain a non-negative adjacency matrix compatible with standard degree-normalized graph convolution.

Our choice of DE PCC (ReLU) was based on empirical comparisons of several spatial prior construction methods, including PCC from raw EEG and DE features with absolute-value or ReLU transformations, as well as PLV. As shown in Table R2, DE PCC (ReLU) achieved the best overall performance among these alternatives, which supports our choice. All evaluated priors improve performance over the model without a spatial prior, supporting the effectiveness of introducing an initial static prior to model inter-electrode relationships.

Table R2: Comparison for PCC construction.

Spatial PriorEAV Accuracy (%) ↑EAV Weighted F1 (%) ↑
No spatial prior40.77 ± 2.0838.82 ± 2.06
Raw EEG PCC (Abs)41.41 ± 3.0140.01 ± 2.83
Raw EEG PCC (ReLU)41.11 ± 2.3240.04 ± 2.46
DE PCC (Abs)41.37 ± 2.8239.75 ± 2.66
PLV41.13 ± 2.5939.78 ± 2.66
DE PCC (ReLU) (Ours)41.49 ± 2.3340.04 ± 1.42

W3 & Q3: Structural Distillation and Injection Design.

Response: We thank the reviewer for these insightful comments.

(1) Why distill S rather than A?

We choose to distill S\mathbf{S} because it directly represents the electrode dependencies learned from AV guidance, whereas A\mathbf{A} further incorporates the static PCC prior and self-connections. The two targets optimize related but conceptually different objectives: distilling S\mathbf{S} isolates the AV-guided component, whereas distilling A\mathbf{A} matches the complete propagation graph.

We also evaluated A\mathbf{A}-level distillation for comparison. Table R3 shows that both targets achieve comparable performance. We conducted a subject-level paired significance test (Table R4). The experiment does not provide evidence that either target is empirically superior; instead, it shows that CDD is robust to this design choice.

We retain S\mathbf{S} as the default because it more precisely isolates the AV-guided knowledge that CDD is designed to transfer.

Table R3: Comparison of matrix S and matrix A as CDD Distillation Targets.

CDD Distillation TargetAccuracy (%)Weighted F1 (%)
Matrix S41.49 ± 2.3340.04 ± 1.42
Matrix A41.50 ± 1.8040.34 ± 1.66

Table R4: Subject-Level Paired Comparison of S- and A-Matrix Distillation on EAV.

Metric95% CI (pp)p-valueA Wins / Ties / Losses
Acc[-0.92, +0.93]0.983713 / 1 / 13
F1[-0.61, +1.24]0.487617 / 0 / 10

(2) Clarification of the Row-wise Injection Method

We would like to clarify the mathematical formulations and mechanisms of the row-wise injection strategy.

Row-wise Injection. Row-wise injection uses the AV-guided resonance score αi(k)\alpha_i^{(k)} to modulate the attention that electrode ii assigns to other electrodes. Specifically,

SrowT,(k)=softmax⁡row((X(k)W1⊤)(X(k)W2⊤)⊤+diag⁡(α(k))(1N1N⊤−IN)τ).\mathbf{S}_{\mathrm{row}}^{T,(k)}=\operatorname{softmax}_{\mathrm{row}}\left(\frac{(\mathbf{X}^{(k)}\mathbf{W}_1^\top)(\mathbf{X}^{(k)}\mathbf{W}_2^\top)^\top+\operatorname{diag}(\boldsymbol{\alpha}^{(k)})(\mathbf{1}_N\mathbf{1}_N^\top-\mathbf{I}_N)}{\tau}\right).

Here, αi(k)\alpha_i^{(k)} is added to all off-diagonal attention scores in the ii-th row. This shifts more attention from the electrode itself to other electrodes when its AV resonance is high. We will add these details to the revised manuscript.

W4 & Q4: Subject-level results and statistical validation. & Dataset selection.

Response: We thank the reviewer for raising this important question.
(1) Subject-level results and statistical validation: We conducted a paired subject-level analysis comparing the performance between EXIST and the strongest baseline(DGCNN on EAV, EEG-SCMM on PME4). The results in Table R5 and R6 demonstrate that our improvements are statistically significant.

As shown in Table R5, across 27 subject–split evaluations, EXIST outperforms DGCNN in 19 cases for accuracy and 18 cases for F1 on the EAV dataset. Paired t-tests confirm the gains are statistically significant (Acc: p = 0.0366; F1: p = 0.0463). This provides statistical evidence that EXIST yields improvements across the strongest baseline method on EAV.

As shown in Table R6, across 6 subject–split evaluations, EXIST outperforms EEG-SCMM in all cases for both Acc and F1 on PME4 dataset. Paired t-tests confirm the gains are statistically significant (Acc: p = 0.0029; F1: p = 0.0343). This provides statistical evidence that EXIST yields improvements across the strongest baseline method on PME4.

These results provide statistical evidence for the effectiveness of EXIST.

(2) Why wasn't the video modality used for the PME4 dataset, and why were the PEGConv and MAHNOB-HCI datasets not chosen for the evaluation:

Although PME4 includes facial video recordings, the dataset only provides pre-extracted visual features from VGG16 for 3 subjects, as not all participants consented to sharing their video data. Therefore, we use audio-only guidance on PME4.

MAHNOB-HCI follows an emotion-elicitation paradigm: participants watch affective videos while their facial responses and EEG are recorded. This is different from our setting, which focuses on actively expressed emotions in more interactive and clinically relevant scenarios. In this elicitation setting, it remains unclear whether facial expressions provide stronger emotional cues than EEG responses. In addition, the audio in MAHNOB-HCI mainly comes from the stimulus videos rather than the participants themselves, so the full audiovisual guidance used in EXIST cannot be directly applied. So we didn't choose it as our evaluation dataset.

PEGConv is very relevant to our setting because it records synchronized EEG, audio, and video during conversations. However, access requires a signed agreement from the data provider, which they don't release. We also contacted the authors but did not receive a response. We therefore could not include PEGConv in the current experiments. If access becomes available, we will evaluate EXIST on this dataset in future work.

Reviewer_2CymOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EXIST, an explicit-guided spatiotemporal knowledge distillation framework for EEG-based emotion recognition. The method leverages audiovisual information during training while requiring only EEG signals at inference. It introduces Cortical Dependency Distillation (CDD) and Emotional Dynamics Distillation (EDD) to transfer spatial and temporal attention patterns from an audiovisual-guided EEG teacher to an EEG-only student.

Experiments on the EAV and PME4 datasets demonstrate improvements over representative EEG-based recognition and knowledge distillation baselines. The paper addresses a relevant problem and presents a technically coherent framework. However, several concerns regarding methodological novelty, experimental reliability, and the interpretation of the learned representations remain insufficiently addressed.

Critical strengths
  1. The paper addresses the practical challenge of leveraging privileged audiovisual information to improve EEG-only emotion recognition under privacy-sensitive deployment settings.

  2. The proposed CDD and EDD modules provide an intuitive approach to transferring spatial and temporal supervisory information without directly matching cross-modal feature representations.

  3. The paper compares against several EEG-based and knowledge distillation methods and includes component ablations, hyperparameter analysis, and additional cross-backbone experiments.

Critical weaknesses

1 The proposed framework mainly combines audiovisual-guided attention, graph-based spatial modeling, temporal attention, and knowledge distillation. While their integration is reasonable, the conceptual distinction from existing attention-transfer and privileged-information distillation approaches is not sufficiently established. The authors should clarify which methodological components are fundamentally new and demonstrate why the proposed supervision mechanism offers advantages beyond existing attention-based distillation strategies. 2 The experiments use only 42 subjects in EAV and 11 subjects in PME4, with three random subject-independent splits. Although multiple training seeds are considered, the small number of independent test subjects raises concerns about the generalizability of the reported improvements. The authors should provide subject-level performance analysis, paired statistical significance tests, and more robust cross-subject evaluations, particularly for PME4. 3 According to Table 3, combining CDD and EDD improves accuracy by only 0.46 percentage points and F1 by 0.05 percentage points over CDD alone. These differences are small relative to the reported variations. Therefore, the current results do not conclusively establish the complementary contributions of the two modules. Additional analysis is needed to determine whether their combination provides a statistically reliable benefit. 4 The proposed spatial prior is constructed using Pearson correlations between five-dimensional differential entropy feature vectors. However, correlations across frequency-band features do not necessarily represent physiological functional connectivity between brain regions. The authors should justify this construction and distinguish learned electrode dependencies from neurophysiologically meaningful cortical connectivity. Additional validation would strengthen the interpretability claims. 5 A central claim is that EEG-native structural distillation mitigates the negative transfer caused by direct cross-modal alignment. Although the reported comparisons support the effectiveness of EXIST, they do not fully isolate the proposed mechanism from other potential explanations. The authors should consider controlled experiments involving shuffled or mismatched audiovisual guidance to verify whether the improvements arise from meaningful cross-modal supervision rather than additional training signals or regularization.

Questions for authors

Q1. Can the authors provide paired statistical significance tests for EXIST and the strongest competing methods across subject-independent splits?

Q2. What is the specific methodological contribution of EXIST beyond conventional attention transfer and privileged-information distillation?

Q3. Given the limited additional improvement of combining CDD and EDD, what evidence supports the claimed complementarity between the two modules?

Q4. How sensitive is the proposed framework to noisy, missing, or mismatched audiovisual information during teacher training?

6 comments

To Reviewer 2Cym

We thank the reviewer for the detailed and constructive feedback. We address each point below.

W1 & Q2: What is the specific methodological contribution of EXIST beyond conventional attention transfer and privileged-information distillation

Response: We thank the reviewer for raising this important question. We would like to clarify that the key contribution of EXIST is not simply a combination of existing techniques, but how to use privileged audiovisual information to construct an effective EEG-native structural supervision, which then serves as the distillation signal for the EEG student.

(1) Beyond conventional privileged-information distillation: Novel EEG-native supervision. EXIST introduces an AV-guided EEG teacher between the AV teacher and EEG student to provide direct EEG-native distillation targets. Most existing privileged-information distillation methods lack this design. They align the AV and EEG modalities directly, even though their representations differ substantially. Table 4 in our paper shows that EEG-native distillation is more effective than directly transferring knowledge from explicit AV modalities to the implicit EEG modality (KD, FitNets, and NST improve from 37.95%, 38.67%, and 36.98% with multimodal teachers to 38.45%, 39.69%, and 38.55% with EEG teachers).

(2) Beyond conventional attention transfer: Novel method for AV-guided attention target construction.
Our main contribution is how to construct more effective EEG-native attention targets from the AV modalities rather than attention distillation itself. EXIST exploits the inherent spatial and temporal characteristics of EEG. Through the bias injection of AV-guided resonance scores, explicit AV information guides the EEG teacher to learn a more informative cortical dependency matrix and temporal dynamics weights, which are more effective targets for structured EEG-native supervision. Table 5 demonstrates the effectiveness of this guidance, improving the EEG teacher's accuracy by 11.61%. Table 4 suggests that our proposed attention distillation outperforms conventional distillation methods even when the baselines use an EEG-native teacher. Furthermore, Table A8 in the appendix suggests that constructing attention targets with explicit AV guidance provides additional performance gains.

W2 & Q1: statistical significance tests for EXIST and the strongest competing methods across subject-independent splits

Response: We thank the reviewer for raising this important question. We conduct a paired subject-level analysis comparing the performance between EXIST and the strongest baseline(DGCNN on EAV, EEG-SCMM on PME4). The results in Table R1 and R2 demonstrate that our improvements are statistically significant.

(1) As shown in Table R1, Across 27 subject–split evaluations, EXIST outperforms DGCNN in 19 cases for accuracy and 18 cases for F1 on EAV dataset. Paired t-tests confirm the gains are statistically significant (Acc: p = 0.0366; F1: p = 0.0463). This provides statistical evidence that EXIST improves on the strongest baseline method on EAV.

(2) As shown in Table R2, Across 6 subject–split evaluations, EXIST outperforms EEG-SCMM in all cases for accuracy and for F1 on PME4 dataset. Paired t-tests confirm the gains are statistically significant (Acc: p = 0.0029; F1: p = 0.0343). This provides statistical evidence that EXIST yields improvements across the strongest baseline method on PME4.

Additional Experiments:

Table R1: Subject-Level Comparison Between EXIST and DGCNN on EAV.

SplitSubjectEXIST Acc. (%)DGCNN Acc. (%)Δ Acc. (pp)EXIST F1 (%)DGCNN F1 (%)Δ F1 (pp)
1S0743.0034.25+8.7538.0334.08+3.95
1S0840.7538.00+2.7535.5432.87+2.67
1S1038.2536.75+1.5029.5429.13+0.41
1S1336.2533.75+2.5027.1124.51+2.60
1S2062.7568.00−5.2559.0264.95−5.93
1S2240.2543.50−3.2531.6138.14−6.53
1S2455.7552.00+3.7553.3151.55+1.76
1S2859.2553.00+6.2555.9250.03+5.89
1S2925.5025.25+0.2515.5014.55+0.95
2S0140.2533.00+7.2535.3727.90+7.47
2S0454.2552.75+1.5049.9448.32+1.62
2S1427.2525.75+1.5023.0616.51+6.55
2S2333.5037.50−4.0030.3933.23−2.84
2S2460.2553.75+6.5059.4251.41+8.01
2S3028.2525.00+3.2521.8919.24+2.65
2S3244.5040.50+4.0038.0131.78+6.23
2S3533.0031.25+1.7532.9528.96+3.99
2S3730.2532.75−2.5027.2524.84+2.41
3S0231.5035.75−4.2523.8828.46−4.58
3S1233.0030.25+2.7528.6824.71+3.97
3S1432.5031.50+1.0023.4226.23−2.81
3S1536.5029.25+7.2532.5922.11+10.48
3S2069.2569.75−0.5068.1569.57−1.42
3S2337.0037.000.0026.8829.27−2.39
3S2458.0057.00+1.0056.6557.29−0.64
3S3132.2526.50+5.7525.0220.00+5.02
3S3937.0041.75−4.7533.3435.35−2.01
—Paired t-test——p = 0.0366——p = 0.0463

Table R2: Subject-Level Comparison Between EXIST and EEG-SCMM on PME4.

SplitSubjectEXIST ACC. (%)EEG-SCMM ACC. (%)Δ ACC. (pp)EXIST F1 (%)EEG-SCMM F1 (%)Δ F1 (pp)
1S0534.1831.82+2.3634.0629.53+4.53
1S0728.5027.75+0.7522.2722.01+0.26
2S0332.0029.50+2.5027.3425.71+1.63
2S0630.0228.11+1.9127.4323.82+3.61
3S0231.0029.75+1.2527.5425.75+1.79
3S1130.5027.25+3.2521.9221.55+0.37
Paired t-test———p = 0.0029——p = 0.0343

W3 & Q3: Additional analysis is needed to determine whether the combination of CDD and EDD provides a statistically reliable benefit.

Response: CDD distills the AV-guided cortical dependencies among EEG electrodes, while EDD distills the temporal attention weights over EEG windows. The two modules thus provide supervision on spatial dependencies and temporal importance, respectively.

We conduct a paired subject-level analysis comparing the performance of CDD+EDD and CDD-only on identical held-out subjects across three splits. As shown in Table R3, Across 27 subject–split pairs, adding EDD improves accuracy in 20/27 and F1 in 20/27 cases. Paired t-tests confirm the gains are statistically significant (Acc: p = 0.0437; F1: p = 0.0135). This provides evidence that the combination of CDD and EDD provides a statistically reliable benefit, despite the relatively small margin accuracy gain.

Table R3: Subject-Level Comparison Between CDD+EDD and CDD-Only on EAV.

SplitSubjectCDD+EDD Acc. (%)CDD Acc. (%)Δ Acc. (pp)CDD+EDD F1 (%)CDD F1 (%)Δ F1 (pp)
1S0743.0042.25+0.7538.0335.19+2.84
1S0840.7540.00+0.7535.5434.13+1.41
1S1038.2541.00−2.7529.5429.73−0.19
1S1336.2536.00+0.2527.1128.37−1.26
1S2062.7563.50−0.7559.0254.97+4.05
1S2240.2541.00−0.7531.6130.89+0.72
1S2455.7557.00−1.2553.3148.56+4.75
1S2859.2558.75+0.5055.9250.10+5.82
1S2925.5025.25+0.2515.5017.89−2.39
2S0140.2538.50+1.7535.3733.26+2.11
2S0454.2553.00+1.2549.9447.59+2.35
2S1427.2526.25+1.0023.0621.69+1.37
2S2333.5032.25+1.2530.3926.90+3.49
2S2460.2557.00+3.2559.4253.71+5.71
2S3028.2527.50+0.7521.8920.76+1.13
2S3244.5043.75+0.7538.0139.05−1.04
2S3533.0031.75+1.2532.9529.60+3.35
2S3730.2528.50+1.7527.2526.82+0.43
3S0231.5030.50+1.0023.8823.32+0.56
3S1233.0033.75−0.7528.6831.02−2.34
3S1432.5031.75+0.7523.4226.53−3.11
3S1536.5037.00−0.5032.5932.18+0.41
3S2069.2569.00+0.2568.1564.03+4.12
3S2337.0036.00+1.0026.8826.22+0.66
3S2458.0057.50+0.5056.6555.79+0.86
3S3132.2532.50−0.2525.0227.24−2.22
3S3937.0036.50+0.5033.3433.03+0.31
—Paired t-test——p = 0.0437——p = 0.0135

W4: Interpretation of the PCC-based spatial prior

Response: We thank the reviewer for this comment. In EXIST, the PCC-based prior is used to construct static correlation between EEG electrodes, rather than to directly estimate physiological brain connectivity. Specifically, DE-based PCC measures the similarity of spectral feature patterns across electrodes, providing an initial graph structure to guide feature aggregation in the subsequent GCN. Our original description of the learned electrode relationships as neurophysiological functional connectivity was not sufficiently precise. We will revise the manuscript to describe them more accurately as statistical electrode relationships and the PCC-based spatial prior.

Our choice of DE-based PCC was informed by prior studies[1] and determined after evaluating several alternative construction strategies.

Table R4 compares different methods for constructing this spatial prior. All evaluated priors improve performance over the model without a spatial prior. Our DE PCC (ReLU) achieves the highest average accuracy. These results support the rationale of our spatial prior design.

Table R4: Comparison for PCC construction on EAV.

Spatial PriorEAV Accuracy (%) ↑EAV Weighted F1 (%) ↑
No spatial prior40.77 ± 2.0838.82 ± 2.06
Raw EEG PCC (Abs)41.41 ± 3.0140.01 ± 2.83
Raw EEG PCC (ReLU)41.11 ± 2.3240.04 ± 2.46
DE PCC (Abs)41.37 ± 2.8239.75 ± 2.66
PLV41.13 ± 2.5939.78 ± 2.66
DE PCC (ReLU) (Ours)41.49 ± 2.3340.04 ± 1.42

Reference
[1] Xu, Qingjie, et al. "A multi-graph convolutional network method for alzheimer’s disease diagnosis based on multi-frequency EEG data with dual-mode connectivity." Frontiers in Neuroscience 19 (2025): 1555657.

Reviewer_S2MVOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper works on the EEG emotion recognition task, where a per-clip discrete label of the subject's emotion needs to be inferred from the EEG signal over the whole clip. The authors tackle the problem with a privileged knowledge distillation approach, employing a teacher model with access to video and audio signals to guide the student model, which has only EEG as input. Instead of distillation in the feature space of the final prediction, the authors encode the video and audio signals into a context vector cc as the explicit stream of the teacher model, which guides the implicit stream, with the same input and architecture as the student model, by injecting a per-feature bias in the softmax function based on the correlation of the context vector and the feature vectors. Results show marginally better accuracy compared with the baselines, and the ablations demonstrate the importance of the explicit guidance.

Critical strengths
  • Comprehensive and Reliable Results Reported. All results across tables and figures report the average and standard deviation over 3 training seeds ×\times 3 dataset split seeds, supporting their reliability. Numerous ablation studies on the hyperparameters are reported in the main paper and the appendix.
  • Consistent Improvement over the Baselines. Despite marginal (discussed below), the results are consistently better than the baselines across metrics and datasets.
Critical weaknesses
  • Insignificant Improvement and Contribution of the EDD component. Despite a consistent improvement, the 1.66% improvement in accuracy of Table 1 is within the standard deviation of ±2.33\pm2.33%, and similarly for other main results reported. In Table 3, which studies the contribution of the proposed components, the addition of EDD only improves by 0.46%, but the error is around 2.5%. It is hard to establish the significance of the EDD module and the entire work based on a marginal improvement.
  • Unclear data for Figure 1 Left. In Figure 1 (left), the accuracies on video, audio, and EEG modalities are shown without an explanation of which metrics or models the authors used to obtain the accuracy numbers. Although it is not an experiment and the authors may use the figure only to demonstrate the difficulty of extracting information from the EEG modality, a figure with metrics in a paper should have proper support. There is a citation; however, I could not find the exact same figure in the cited paper.
  • Overclaimed Gap Removal. The authors claim that the modality gap is resolved by design. With the implicit stream whose architecture is the same as the student, I can admit that the gap is mitigated, but I do not believe it is sufficient to claim that the gap is resolved since the teacher model still receives information through the α\alpha injection.
Questions for authors
  • In the caption and the "Ours" cell of Table 1, there is a dagger sign, but I could not find the corresponding footnote. Can the authors explain what they intended to note with the dagger sign?
  • For the baselines in Tables 1 and 2, which of them are re-implemented by the authors, and which of them report the numbers in the original publication or model weights published online? For those that the authors re-implemented, how were the hyperparameters selected and tuned to ensure a fair comparison?
7 comments

To Reviewer S2MV

We really appreciate the reviewer for the detailed and constructive feedback. We carefully addressed the concerns and questions below.

W1: Statistical significance and the contribution of EDD. Question about the significance of the gains: the 1.66% improvement in Table 1 and the 0.46% gain from EDD in Table 3 both fall within the reported standard deviation.

Response: We thank the reviewer for raising this important concern regarding the improvements and the contribution of EDD. We conducted paired subject-level analyses with paired t-tests for both CDD+EDD vs. CDD-only and EXIST vs. DGCNN (the strongest baseline). The results in Table R1 and R2 demonstrate that our improvements are statistically significant.

We also conducted a temporal-window perturbation experiment by masking windows with the highest and lowest attention weights to verify that EDD captures emotion-relevant temporal windows. The results in Table R3 demonstrate the effective contribution of EDD to emotion recognition.

(1) Statistical Significance of the Improvements.
CDD+EDD and CDD-only: As shown in Table R1, Across 27 subject–split pairs, adding EDD improves accuracy in 20/27 and F1 in 20/27 cases. Paired t-tests show the gains are statistically significant (Acc: p = 0.0437; F1: p = 0.0135). This provides statistical evidence that EDD yields improvements across subjects, despite the relatively small margin accuracy gain.

EXIST and the strongest baseline DGCNN: As shown in Table R2, across 27 subject–split evaluations, EXIST outperforms DGCNN in 19 cases for accuracy and 18 cases for F1. Paired t-tests show the gains are statistically significant (Acc: p = 0.0366; F1: p = 0.0463). This provides statistical evidence that EXIST yields improvements across the strongest baseline method, despite the relatively 1.66 accuracy gain.

(2) Contribution of EDD. EDD aims to transfer AV-guided temporal dynamics to the EEG student, encouraging it to focus on temporally informative EEG windows for emotion recognition. As shown in Table R3, masking high-attention windows consistently leads to larger drops in accuracy and F1 than masking low-attention windows under both 1 and 2 window settings, with more differences when two windows are masked (3.80/3.60 vs. 1.94/2.22 pp). These results support the effectiveness of EDD in guiding the student to emphasize more informative temporal windows.

Table R3. Temporal-Window Perturbation Analysis on EAV

Masked WindowsHigh-attention Acc. (%)High-attention Weighted F1 (%)Low-attention Acc. (%)Low-attention Weighted F1 (%)
041.49 ± 2.3340.04 ± 1.4241.49 ± 2.3340.04 ± 1.42
140.47 ± 2.7739.35 ± 2.2341.01 ± 1.9339.44 ± 0.99
237.69 ± 2.7736.44 ± 2.7739.55 ± 2.4037.82 ± 1.28

Overall, these results demonstrate the effectiveness of EXIST for EEG-based emotion recognition and further highlight the contribution of EDD in transferring AV-guided temporal information.

Additional Experiments:

Table R1: Subject-Level Comparison Between CDD+EDD and CDD-Only on EAV.

SplitSubjectCDD+EDD Acc. (%)CDD Acc. (%)Δ Acc. (pp)CDD+EDD F1 (%)CDD F1 (%)Δ F1 (pp)
1S0743.0042.25+0.7538.0335.19+2.84
1S0840.7540.00+0.7535.5434.13+1.41
1S1038.2541.00−2.7529.5429.73−0.19
1S1336.2536.00+0.2527.1128.37−1.26
1S2062.7563.50−0.7559.0254.97+4.05
1S2240.2541.00−0.7531.6130.89+0.72
1S2455.7557.00−1.2553.3148.56+4.75
1S2859.2558.75+0.5055.9250.10+5.82
1S2925.5025.25+0.2515.5017.89−2.39
2S0140.2538.50+1.7535.3733.26+2.11
2S0454.2553.00+1.2549.9447.59+2.35
2S1427.2526.25+1.0023.0621.69+1.37
2S2333.5032.25+1.2530.3926.90+3.49
2S2460.2557.00+3.2559.4253.71+5.71
2S3028.2527.50+0.7521.8920.76+1.13
2S3244.5043.75+0.7538.0139.05−1.04
2S3533.0031.75+1.2532.9529.60+3.35
2S3730.2528.50+1.7527.2526.82+0.43
3S0231.5030.50+1.0023.8823.32+0.56
3S1233.0033.75−0.7528.6831.02−2.34
3S1432.5031.75+0.7523.4226.53−3.11
3S1536.5037.00−0.5032.5932.18+0.41
3S2069.2569.00+0.2568.1564.03+4.12
3S2337.0036.00+1.0026.8826.22+0.66
3S2458.0057.50+0.5056.6555.79+0.86
3S3132.2532.50−0.2525.0227.24−2.22
3S3937.0036.50+0.5033.3433.03+0.31
—Paired t-test——p = 0.0437——p = 0.0135

Table R2: Subject-Level Comparison Between EXIST and DGCNN on EAV.

SplitSubjectEXIST Acc. (%)DGCNN Acc. (%)Δ Acc. (pp)EXIST F1 (%)DGCNN F1 (%)Δ F1 (pp)
1S0743.0034.25+8.7538.0334.08+3.95
1S0840.7538.00+2.7535.5432.87+2.67
1S1038.2536.75+1.5029.5429.13+0.41
1S1336.2533.75+2.5027.1124.51+2.60
1S2062.7568.00−5.2559.0264.95−5.93
1S2240.2543.50−3.2531.6138.14−6.53
1S2455.7552.00+3.7553.3151.55+1.76
1S2859.2553.00+6.2555.9250.03+5.89
1S2925.5025.25+0.2515.5014.55+0.95
2S0140.2533.00+7.2535.3727.90+7.47
2S0454.2552.75+1.5049.9448.32+1.62
2S1427.2525.75+1.5023.0616.51+6.55
2S2333.5037.50−4.0030.3933.23−2.84
2S2460.2553.75+6.5059.4251.41+8.01
2S3028.2525.00+3.2521.8919.24+2.65
2S3244.5040.50+4.0038.0131.78+6.23
2S3533.0031.25+1.7532.9528.96+3.99
2S3730.2532.75−2.5027.2524.84+2.41
3S0231.5035.75−4.2523.8828.46−4.58
3S1233.0030.25+2.7528.6824.71+3.97
3S1432.5031.50+1.0023.4226.23−2.81
3S1536.5029.25+7.2532.5922.11+10.48
3S2069.2569.75−0.5068.1569.57−1.42
3S2337.0037.000.0026.8829.27−2.39
3S2458.0057.00+1.0056.6557.29−0.64
3S3132.2526.50+5.7525.0220.00+5.02
3S3937.0041.75−4.7533.3435.35−2.01
—Paired t-test——p = 0.0366——p = 0.0463

W2: Unclear accuracy values in Figure 1 (left).

Response: We thank the reviewer for pointing out this lack of clarity. The accuracies in Figure 1 (left) were obtained on the EAV dataset using a random experimental split, yielding 54.97% for audio, 45.34% for video, and 39.50% for EEG. The audio and video results were obtained using BiLSTM models with features extracted by openSMILE and OpenFace, respectively. The EEG results were obtained using our EEG student model without distillation. The figure aims to illustrate the greater difficulty of EEG-based emotion recognition relative to audiovisual modalities. We will replace the results in the revised manuscript with the data obtained across all experimental splits: Audio: 50.61 ± 0.63%, Video: 44.11 ± 2.65%, EEG: 38.72 ± 2.58% and clarify the value reference.

W3: Overstatement regarding the modality gap.

Response: Our intention was to emphasize that EXIST introduces an AV-guided EEG teacher to enable EEG-native knowledge distillation, avoiding direct alignment between heterogeneous AV and EEG representations and thereby mitigating the modality gap. We will replace "resolved" with "mitigated" in the revised manuscript.

Q1: Meaning of the dagger symbol in Table 1.

Response: We would like to clarify that the dagger (†) was defined in the caption of Table 1 as indicating knowledge distillation from explicit modalities during training. It is intended to distinguish EXIST from EEG-only methods without knowledge distillation.

Q2: Baseline implementation details.

Response: Thank you for raising this important question. None of the baselines had reported cross-subject results on EAV or PME4. Our setting targets a more practical scenario: active emotional expression with privacy-preserving EEG-only inference. Therefore, we retrained all methods on both datasets.

(1) For a fair comparison, in Table 1, We follow the official implementations of all EEG-only baselines, including their architectures and input representations.
Two models have been adapted and modified according to the tasks. We adapted iTransformer to EEG classification by embedding each channel’s entire time series as a single token, applying self-attention across channels with a learnable channel embedding, and replacing the forecasting projection with a classification head. We initialized LaBraM using the publicly released LaBraM-Base checkpoint, and fine-tuned the model on each training split for 50 epoch.
For methods involving temporal segmentation, we adopt the same 1-second non-overlapping windowing scheme as EXIST to ensure a fair comparison.

(2) For all distillation baselines in Table 2, we used the same EEG student and multimodal teacher architectures as EXIST (GCN, BiLSTM) to ensure fair comparison. Feature-based methods align the student’s attention-pooled BiLSTM representation before the classification head with the corresponding multimodal teacher feature. All other training settings were kept identical to EXIST to ensure a fair comparison.
These settings ensure that our comparison with the baselines is fair.

For hyperparameter tuning, we kept the model-architectural hyperparameters of each baseline from the official implementations. The learning rate was searched from 1e-3 to 1e-5 on a linear scale, while the batch size was searched over powers of two from 8 to 256. The best configuration was selected based on validation performance, with early stopping applied.

We hope the response has addressed your concerns. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.