acceptodds
Under review as a conference paper at ICLR 2027

A Unified Framework for EEG–Video Emotion Recognition with Brain Anatomy and Temporal Priors

Abstract

Recent studies in video- and EEG-based emotion recognition have shown notable progress. However, multi-modal emotion recognition remains largely unexplored, particularly the integration of physiological signals with video. This integration is crucial, as EEG–video fusion combines observable behavioral cues with internal neural dynamics and enables a more comprehensive and robust characterization of human emotion. To this end, we propose EVER, a novel EEG–Video Emotion Recognition framework that effectively integrates complementary information from both modalities. Specifically, EVER models the anatomical organization of multi-channel EEG signals and the temporal relationships among video clips through graph-based representations. These structured features are subsequently integrated with global modality representation using an inter-modal graph for emotion prediction. To provide a comprehensive evaluation, we establish a benchmark across two public EEG-video paired datasets, Emognition and MDMER. We evaluate 12 representative models, consisting of 5 EEG-only, 5 video-only, and 2 EEG-video models. Extensive experiments demonstrate that the proposed EVER achieves state-of-the-art performance by jointly modeling behavioral cues from video and physiological responses, thereby enabling the recognition of emotional patterns unattainable by either modality alone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
EEG–Video Emotion Recognition, Benchmark, Graph Neural Networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qnJ1OpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

EVER addresses EEG–video emotion recognition. It argues that existing fusion methods pool each modality into one vector, discarding the anatomical organization of EEG channels and the temporal evolution of facial behavior. EVER keeps these fine-grained structures and fuses them with pretrained global representations in one graph.

Contributions.

BAR-GCN: per-channel EEG spectrogram embeddings are connected by a GCN only within 10–20 scalp regions, then pooled into region nodes. Mapping to standard regions lets different electrode montages share the scheme. VCG: video clip embeddings from VideoMAE or AdaMAE become temporally ordered nodes, with sinusoidal position codes and a learnable Gaussian that weakens edges between distant clips. Inter-modal graph: connects the clip nodes, the global CBraMod EEG embedding and the region nodes under a structured edge mask, followed by a 2-layer GCN and a per-class gate between the EEG and video outputs. Benchmark: 12 unimodal and multimodal models compared on Emognition and MDMER under subject-independent splits.

Validation.

Main results: 5-class valence and arousal classification. EVER reaches 38.6% / 35.5% average accuracy, against 35.3% (CBraMod) and 33.4% (VideoMAE) for the best baselines. Ablations: components, anatomical vs. random region partitions, VCG parts and fusion operators. Appendix analyses: bootstrap confidence intervals, EEG-noise robustness, model size, modality complementarity and additional EEG encoders.

Critical strengths

A sensible fusion design for an underexplored pairing. Rather than pooling each modality, EVER keeps EEG regions and ordered video clips as separate nodes and lets them interact with the pretrained global embedding. Mapping electrodes to canonical scalp regions also lets the same model handle different montages (4 and 18 channels here), which matters for a field where every dataset uses a different headset. More statistical candor than is typical for the area. All models share one subject-independent split, loss and training budget. The authors report bootstrap confidence intervals and explicitly say the differences on Emognition are not statistically separated. That honesty makes the paper easier to assess, even though it also exposes how thin the evidence is.

Critical weaknesses

The evaluation can’t distinguish the gains from noise. Emognition’s test set is 88 samples from 8 subjects, and there is a single subject split and a single training run. EVER’s +3.3-point lead over CBraMod amounts to about three samples per target, inside the paper’s own confidence intervals. Early stopping uses a “validation score,” but only train and test splits are described. If the test set was used for model selection, every number is optimistically biased. No majority-class baseline is given for the 5-way binned labels, so it’s unclear how far above trivial the 24–39% accuracies are. For a result at this scale to be convincing, it needs leave-one-subject-out or repeated subject splits with variance across seeds. The ablations and baselines behave in ways that undermine the claimed mechanism. Removing only the positional encoding or only the Gaussian kernel drops EVER to 30.7% and 29.0% (Table 4), below the EEG-only CBraMod model (35.3%) that sits inside EVER. All four fusion baselines in Table 5 (concatenation, cross-attention, FiLM, pooled-node GCN) also fall below EEG alone. Fusion that makes a model worse than its strongest input points to unstable training or poorly tuned baselines, not to the graph structure being essential. The complementarity analysis also shows the fused model mostly follows video: on MDMER arousal, it keeps only 13 of the 111 samples that EEG alone gets right. A simple logit-averaging ensemble of the same two backbones is missing, and it’s the baseline most likely to show whether the graph machinery is needed at all.

Questions for authors

Which set is used for early stopping? If it’s not the test set, what are its size and subjects? Can you report leave-one-subject-out or repeated subject-split results (mean ± std) over several seeds? What are the majority-class and chance accuracies per target after 5-way binning? How does a logit-averaging ensemble of the same CBraMod and VideoMAE backbones perform? Why do all fusion baselines in Table 5, and the single-component removals in Table 4, fall below the EEG-only model?

8 comments
AuthorsOpenReview ↗

Response to Reviewer qnJ1 (1/3) - Task context and early stopping (Q1)

We first apologize that the number of evaluation samples in the submission was incorrect because of an implementation issue in the distributed evaluation. Please kindly refer to our response to Reviewer k3Z3 (W1(a) and Q1) for the details. Accordingly, the results in this response are updated to the unique test trials and may differ slightly.

We thank the reviewer for acknowledging the fusion design, the montage-flexible region mapping and the statistical candor of the paper. In brief, we add (i) an evaluation with held-out validation subjects for Q1, (ii) repeated runs over three seeds for Q2, (iii) the majority-class and chance levels for Q3, (iv) a logit-averaging ensemble of the same backbones for Q4, and (v) a fusion comparison with identical backbones on MDMER for Q5. The questions cover the points raised in the weaknesses, so we respond in the order of the questions and address the remaining point on the complementarity analysis at the end.

Context on the task

Subject-independent emotion recognition from EEG and facial video differs from a standard vision task in three ways. First, the labels are self-reported valence and arousal ratings of each trial, which are subjective and vary in scale across participants. Second, EEG shows strong inter-subject variability, so a model must generalize to unseen people rather than to unseen images. Third, participants watch the stimuli passively and often show only subtle facial expressions, and the paired EEG–video datasets are small, with about 400 trials in Emognition and 2.3K trials in MDMER in total. For these reasons, accuracies on five-class valence and arousal remain modest even for large pretrained models. In our benchmark, the EEG foundation models CBraMod, LaBraM, REVE and BIOT all stay between 0.25 and 0.36 average accuracy. A method should thus be assessed by its margin over the trivial and the strongest unimodal baselines on identical samples.

Q1. Set used for early stopping

Every model in the benchmark is reported at its best epoch on the evaluation split, under one identical protocol for all 10 unimodal backbones on both datasets and for all fusion models. The video encoders VideoMAE and AdaMAE are pretrained by self-supervised masked autoencoding on large unlabeled video collections, and only their final encoder block and the classification head are fine-tuned (Section 3.1.1). The EEG foundation models are pretrained on large EEG corpora and fine-tuned with their downstream setups. In the subject-independent setting, the models start to overfit the training subjects within a few epochs, and the evaluation loss rises thereafter, so we reported the best epoch identically for every model.

To examine whether this selection creates the advantage of EVER, we separate the model selection from the test subjects. Eight validation subjects (256 trials) are held out from the training split and are used for model selection. The test subjects are identical to the main paper and are used only for the final evaluation. The evaluation is conducted on MDMER, where the larger number of subjects permits a validation set of this size. It covers EVER and its two unimodal backbones, CBraMod and VideoMAE. The results over three seeds are shown in Tab. R1.

Table R1. Evaluation with held-out validation subjects on the MDMER dataset (mean ± standard deviation over three seeds).

ModelAccUARW-F1
CBraMod (EEG)0.245 ± 0.0170.201 ± 0.0090.211 ± 0.016
VideoMAE (video)0.259 ± 0.0130.212 ± 0.0110.247 ± 0.015
EVER (VideoMAE + CBraMod)0.266 ± 0.0130.221 ± 0.0110.253 ± 0.005

This protocol is stricter than the main paper, because the model selection is performed without access to the test subjects. Compared with the three-seed results selected on the evaluation split in Tab. R2, the accuracy decreases by a similar amount for both models, by 0.055 for EVER and by 0.050 for CBraMod. This indicates that the selection does not create the gap between them. EVER remains higher than both of its unimodal backbones in all three metrics. The gain is clearest on UAR and W-F1, the two metrics that account for class imbalance. For instance, W-F1 reaches 0.253 against 0.211 for CBraMod and 0.247 for VideoMAE, and UAR reaches 0.221 against 0.201 and 0.212. We will include this result in the main manuscript.

AuthorsOpenReview ↗

Response to Reviewer qnJ1 (2/3) - Repeated runs, majority baseline and ensemble (Q2, Q3, Q4)

Q2. Leave-one-subject-out or repeated runs over seeds

Although the evaluation with held-out validation subjects under Q1 is already reported over three seeds, we also report repeated runs of EVER and CBraMod under the original protocol of the main paper. To show both backbone combinations of EVER effectively, EVER uses AdaMAE + CBraMod on Emognition and VideoMAE + CBraMod on MDMER. The results over three seeds are shown in Tab. R2. EVER is higher on all three metrics on both datasets. The gain is clearest on UAR and W-F1, the two metrics that account for class imbalance. UAR rises from 0.283 to 0.310 on Emognition and from 0.209 to 0.225 on MDMER. W-F1 rises from 0.272 to 0.301 on Emognition and from 0.234 to 0.269 on MDMER, a relative improvement of 11% and 15%. This indicates that EVER improves the recognition across the emotion classes, not only that of the frequent ones. Acc shows some variation across runs, and its mean remains higher for EVER. The advantage of EVER is not specific to the single run reported in the main paper, and it holds on average over repeated runs on both datasets despite their small test sets.

Table R2. Mean ± standard deviation over three seeds (Acc / UAR / W-F1).

ModelEmognition datasetMDMER dataset
CBraMod0.322 ± 0.030 / 0.283 ± 0.036 / 0.272 ± 0.0330.295 ± 0.009 / 0.209 ± 0.003 / 0.234 ± 0.014
EVER0.341 ± 0.026 / 0.310 ± 0.013 / 0.301 ± 0.0080.321 ± 0.029 / 0.225 ± 0.014 / 0.269 ± 0.021

Leave-one-subject-out evaluation would require one training run per held-out subject for every model on both datasets, which is not feasible within the rebuttal period. The paper also reports bootstrap confidence intervals in Tab. A3 of the appendix. Extending this analysis, a paired bootstrap on identical MDMER test trials shows that EVER exceeds CBraMod with statistically significant differences in all three metrics, as shown in Tab. R3.

Table R3. Paired comparison on the MDMER dataset over the 475 unique test trials (two-sided paired-bootstrap p-value of the difference from EVER in parentheses).

ModelAcc (p-value)UAR (p-value)W-F1 (p-value)
EVER (VideoMAE + CBraMod)0.3540.2370.280
CBraMod (best EEG)0.302 (0.012)0.208 (0.007)0.219 (0.001)
Logit-averaging ensemble of CBraMod and VideoMAE0.306 (0.019)0.208 (0.011)0.217 (0.001)

Q3. Majority-class and chance levels

With five classes per target, uniform chance is 0.20 in accuracy. The majority-class predictor always outputs the most frequent class of the test set for each target, the strongest input-independent prediction. Its accuracy equals the share pp of that class. Its UAR is 0.20 because one class has recall 1 and the other four have recall 0. Its W-F1 is 2p2/(1+p)2p^2/(1+p) because only the predicted class has a nonzero F1 score. As for every model in the paper, each metric is computed per target and then averaged over valence and arousal. The results are shown in Tab. R4. EVER with VideoMAE + CBraMod exceeds this predictor on each target of both datasets, by 7.1 points on Emognition and by 3.7 points on MDMER in average accuracy, and also in UAR and W-F1.

Table R4. Majority-class predictor and EVER (VideoMAE + CBraMod).

Emognition datasetMDMER dataset
Majority-class Acc (valence / arousal)0.282 / 0.3180.335 / 0.299
Majority-class average (Acc / UAR / W-F1)0.300 / 0.200 / 0.1390.317 / 0.200 / 0.153
EVER Acc (valence / arousal)0.388 / 0.3530.377 / 0.331
EVER average (Acc / UAR / W-F1)0.371 / 0.353 / 0.3530.354 / 0.237 / 0.280

Q4. Logit-averaging ensemble

We added the logit-averaging ensemble of the same backbones as EVER for both backbone combinations on both datasets. The results are shown in Tab. R5. EVER outperforms the ensemble in all three metrics for both combinations on both datasets, and on MDMER with VideoMAE + CBraMod the gains are statistically significant in all three under the paired bootstrap, as shown in Tab. R3. This shows that the graph-based fusion contributes beyond combining two strong backbones.

Table R5. Logit-averaging ensemble versus EVER with the same backbones (Acc / UAR / W-F1, over the unique test trials of each dataset).

BackbonesModelEmognition datasetMDMER dataset
CBraMod + AdaMAEEnsemble0.353 / 0.288 / 0.2940.300 / 0.203 / 0.212
CBraMod + AdaMAEEVER0.371 / 0.306 / 0.2980.341 / 0.231 / 0.274
CBraMod + VideoMAEEnsemble0.359 / 0.292 / 0.2980.306 / 0.208 / 0.217
CBraMod + VideoMAEEVER0.371 / 0.353 / 0.3530.354 / 0.237 / 0.280
AuthorsOpenReview ↗

Response to Reviewer qnJ1 (3/3) - Fusion baselines and complementarity (Q5)

Q5. Why fusion baselines and single-component removals fall below EEG-only

On Emognition, EEG is far stronger than video (0.353 for CBraMod versus at most 0.307 for the video models in Tab. 1 of the main paper), and fusing the weaker modality without structure pulls the prediction toward it. All fusion variants fine-tune the same large self-supervised backbones with the same loss and hyperparameters as EVER, and the same recipe applied to each backbone alone yields the strongest unimodal models of the benchmark, CBraMod among the EEG models and VideoMAE and AdaMAE among the video models. The degradation therefore comes from the fusion operators rather than from the optimization. The Emognition comparison is reported in Tab. 5 of the main paper. On MDMER, concatenation, cross-attention and FiLM collapse to the majority-class predictor for both video backbones, whereas EVER does not, as shown in Tab. R6.

Table R6. Fusion strategies with identical backbones on the MDMER dataset.

Fusion (Acc / UAR / W-F1)AdaMAE + CBraModVideoMAE + CBraMod
Majority-class predictor0.317 / 0.200 / 0.1530.317 / 0.200 / 0.153
Concatenation0.317 / 0.200 / 0.1530.317 / 0.200 / 0.153
Cross-attention0.317 / 0.200 / 0.1530.317 / 0.200 / 0.153
FiLM0.317 / 0.200 / 0.1530.317 / 0.200 / 0.153
Pooled-node GCN0.320 / 0.202 / 0.1670.296 / 0.195 / 0.156
EVER (ours)0.341 / 0.231 / 0.2740.354 / 0.237 / 0.280

This also shows why each component of EVER matters. Vector-level fusion mixes a strong and a weak modality in a single step, so the weaker one dilutes the stronger one. EVER instead turns each modality into structured nodes, with BAR-GCN producing anatomical region nodes and VCG producing temporally ordered clip nodes. The inter-modal graph lets every clip exchange information with the global EEG embedding and the region nodes under the edge mask, so that the video evidence refines the EEG representation instead of diluting it. Removing the positional encoding or the Gaussian prior takes away the temporal structure of the clip nodes, which explains the drops in Tab. 4 of the main paper.

Complementarity analysis (raised in the weaknesses)

The reviewer notes that EVER keeps only 13 of the 111 arousal samples that EEG alone classifies correctly. The predictions in Fig. A1 were taken from a different evaluation run of the unimodal baselines than the one in Tab. 1 of the main paper, and the figure will be regenerated. On MDMER arousal, the video-only model concentrates almost all of its predictions on the majority class, whereas EVER predicts four distinct classes, so the fused model does not simply follow the video prediction.

Reviewer_YUsQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EVER, a unified cross-modal graph framework for EEG–video emotion recognition, which fuses video clip, global EEG, and brain-region nodes through a unified graph structure, and introduces brain anatomy-aware regional graph modeling BAR-GCN and a video clip Gaussian temporal prior VCG to replace simple global pooling and concatenation. The paper also establishes subject-independent benchmarks on two time-synchronized paired EEG–video datasets and conducts systematic ablation studies, confidence interval analysis, robustness tests, and modality complementarity analysis.

Critical strengths

The paper proposes the EVER unified cross-modal graph framework, which integrates brain anatomical perception regions modeling, video clip Gaussian temporal priors, and masked fusion graphs in a relatively natural manner, and designs a holistic approach. Under a unified setting, subject-independent benchmark tests are conducted, supplemented by ablation studies, confidence intervals, robustness, and modality complementarity analyses, covering a comprehensive range. The paper is written clearly, with sufficient reproducible details, and the module parameter overhead is relatively small.

Critical weaknesses
  1. The novelty positioning is insufficient, and the design concept of the designed modules is similar to the existing methods.
  2. The statistical evidence is insufficient. The Emognition sample size is small, the confidence interval is wide, and the MDMER differences are not significant with limited gains.
  3. The mechanism is not well clarified. There is no clip-level time alignment between clip and EEG, and the VCG time structure does not interact with the EEG time dynamics.
Questions for authors
  1. The novelty positioning is insufficient, and the design concept of the designed modules is similar to the existing methods.
  2. The statistical evidence is insufficient. The Emognition sample size is small, the confidence interval is wide, and the MDMER differences are not significant with limited gains.
  3. The mechanism is not well clarified. There is no clip-level time alignment between clip and EEG, and the VCG time structure does not interact with the EEG time dynamics.
7 comments
AuthorsOpenReview ↗

Response to Reviewer YUsQ (1/2) - Novelty and statistical evidence (W1, W2, Q1, Q2)

We first apologize that the number of evaluation samples in the submission was incorrect because of an implementation issue in the distributed evaluation. Please kindly refer to our response to Reviewer k3Z3 (W1(a) and Q1) for the details. Accordingly, the results in this response are updated to the unique test trials and may differ slightly.

We thank the reviewer for the positive assessment of the clarity and design of EVER. In brief, we (i) clarify the positioning of EVER against existing methods for W1, supported by a fusion comparison with identical backbones, (ii) add a paired significance test and repeated runs over three seeds for W2, and (iii) clarify the trial-level EEG encoding for W3.

W1 and Q1. Novelty and positioning

The contribution of EVER is a complete EEG–video framework together with the first comprehensive benchmark of strong pretrained backbones for this task, rather than a single module. To our knowledge, no prior work has paired video transformers with EEG foundation models for this task. We evaluate five video and five EEG models on both datasets under one subject-independent protocol.

Each component also differs from prior designs. Regional EEG graphs [1–3] are tied to the electrode layout of a specific montage, whereas BAR-GCN maps electrodes to the canonical regions of the 10–20 system and handles the 4-channel and 18-channel headsets with the same design. Learnable Gaussian kernels [4] have been used as temporal convolutions, whereas VCG uses them to weight the edges among pretrained clip nodes. Unlike existing EEG–video methods [5, 6], which fuse globally pooled features or predictions, EVER connects the global EEG embedding, the region nodes and the clip nodes in one inter-modal graph.

The gain of EVER comes from this graph integration. With identical backbones, loss and hyperparameters, the standard fusion operators remain at the level of the majority-class predictor on MDMER for both video backbones, while EVER exceeds it in all three metrics, as shown in Tab. R1. This predictor outputs the most frequent test class of each target.

Table R1. Fusion strategies with identical backbones on the MDMER dataset.

Fusion (Acc / UAR / W-F1)AdaMAE + CBraModVideoMAE + CBraMod
Majority-class predictor0.317 / 0.200 / 0.1530.317 / 0.200 / 0.153
Concatenation0.317 / 0.200 / 0.1530.317 / 0.200 / 0.153
Cross-attention0.317 / 0.200 / 0.1530.317 / 0.200 / 0.153
FiLM0.317 / 0.200 / 0.1530.317 / 0.200 / 0.153
Pooled-node GCN0.320 / 0.202 / 0.1670.296 / 0.195 / 0.156
EVER (ours)0.341 / 0.231 / 0.2740.354 / 0.237 / 0.280

W2 and Q2. Statistical evidence

Tab. A3 of the appendix reports a marginal bootstrap interval for each model, and some of these intervals overlap on MDMER. Such overlap does not imply a non-significant difference, because all models are evaluated on the same test trials. We therefore test the differences directly with a paired bootstrap. As shown in Tab. R2, EVER outperforms the strongest EEG model, CBraMod, with statistically significant differences in all three metrics. The gain is also substantial, as W-F1 rises from 0.219 to 0.280, a relative improvement of 28%, and accuracy moves from below the majority-class level (0.317) to above it. The logit-averaging ensemble of the same backbones is also significantly below EVER in all three metrics, so the gain comes from the graph-based fusion rather than from adding a second backbone.

Table R2. Paired comparison on the MDMER dataset over the 475 unique test trials (two-sided paired-bootstrap p-value of the difference from EVER in parentheses).

ModelAcc (p-value)UAR (p-value)W-F1 (p-value)
EVER (VideoMAE + CBraMod)0.3540.2370.280
CBraMod (best EEG)0.302 (0.012)0.208 (0.007)0.219 (0.001)
Logit-averaging ensemble of CBraMod and VideoMAE0.306 (0.019)0.208 (0.011)0.217 (0.001)

The concern about the small Emognition test set is addressed with repeated runs in the next comment (2/2).

AuthorsOpenReview ↗

Response to Reviewer YUsQ (2/2) - Repeated runs and temporal alignment (W2, Q2, W3, Q3)

W2 and Q2 (continued). Repeated runs

The paired test in Tab. R2 compares the models on the MDMER test trials with one trained model each. We agree that the Emognition test set is small, and the paper does not claim statistical separation there. To examine whether the gain also holds across training runs, we add repeated runs on both datasets. Within the rebuttal period, our compute limits them to EVER and CBraMod, which is the EEG-only counterpart of EVER and is light enough to retrain repeatedly. To show both backbone combinations of EVER effectively, EVER uses AdaMAE + CBraMod on Emognition and VideoMAE + CBraMod on MDMER. The results over three seeds are shown in Tab. R3.

Table R3. Mean ± standard deviation over three seeds (Acc / UAR / W-F1).

ModelEmognition datasetMDMER dataset
CBraMod0.322 ± 0.030 / 0.283 ± 0.036 / 0.272 ± 0.0330.295 ± 0.009 / 0.209 ± 0.003 / 0.234 ± 0.014
EVER0.341 ± 0.026 / 0.310 ± 0.013 / 0.301 ± 0.0080.321 ± 0.029 / 0.225 ± 0.014 / 0.269 ± 0.021

EVER is higher on all three metrics on both datasets. The gain is clearest on UAR and W-F1, the two metrics that account for class imbalance. UAR rises from 0.283 to 0.310 on Emognition and from 0.209 to 0.225 on MDMER. W-F1 rises from 0.272 to 0.301 on Emognition and from 0.234 to 0.269 on MDMER, a relative improvement of 11% and 15%. This indicates that EVER improves the recognition across the emotion classes, not only that of the frequent ones. The advantage of EVER is not specific to the single run reported in the main paper, and it holds on average over repeated runs on both datasets despite their small test sets. Together with the paired test, these results strengthen the statistical evidence for the gain of EVER.

W3 and Q3. Temporal alignment between clips and EEG

EEG is encoded at the trial level by design. Each label describes the complete stimulus interval rather than individual clips (Section 3.1.2), so a clip-level alignment would have no supervision. The EEG branch also resamples the complete trial to a fixed input of ten 200-sample patches for CBraMod, so the segment that corresponds to one 4 s clip would typically span fewer than two patches, which is far below the context of its pretraining. The temporal structure of VCG still interacts with the EEG, because every clip node is connected to the global EEG node and to all region nodes in the inter-modal graph (Section 3.4). We will clarify this in the revision.

References

[1] Ye et al., Hierarchical dynamic graph convolutional network with interpretability for EEG-based emotion recognition. IEEE TNNLS 2022.

[2] Jin et al., PGCN: Pyramidal graph convolutional network for EEG emotion recognition. IEEE TMM 2024.

[3] Du et al., A multi-dimensional graph convolution network for EEG emotion recognition. IEEE TIM 2022.

[4] Piergiovanni and Ryoo, Temporal Gaussian mixture layer for videos. ICML 2019.

[5] Lee et al., EAV: EEG-audio-video dataset for emotion recognition in conversational contexts. Scientific Data 2024.

[6] Wang et al., Milmer: A framework for multiple instance learning based multimodal emotion recognition. Neurocomputing 2026.

AuthorsOpenReview ↗

Summary of our response and the revised manuscript

We are grateful to the reviewer for the careful evaluation of our submission. Our response clarifies the positioning of EVER and adds a fusion comparison with identical backbones for W1, a paired comparison and repeated runs for W2, and a clarification of the trial-level design of the EEG branch for W3.

Regarding the statistical evidence in W2, the revised manuscript additionally reports an evaluation with held-out validation subjects over five seeds together with a paired bootstrap (Sec. 4.3, Tab. 2), which was conducted during the discussion with the other reviewers. These points are summarized in our general comment.

Since only a few hours remain in the discussion period, we would greatly appreciate it if the reviewer could let us know whether any questions remain.

Reviewer_k3Z3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EVER, a framework for subject-independent emotion recognition using facial video and EEG recorded simultaneously while participants watch video stimuli. It aims to combine complementary information from facial behavior and brain activity, addressing the limitations of facial cues under suppressed expressions or poor lighting and the sensitivity of EEG to noise and individual differences.

The targets are participants’ self-reported valence and arousal scores, collected after each stimulus on a nine-point scale. These scores are mapped to five classes: {1,2}, {3,4}, {5}, {6,7}, and {8,9}. Labels are assigned to entire stimulus trials rather than individual video clips, and evaluation subjects are excluded from training.

The authors identify three limitations of existing EEG–video fusion approaches: loss of electrode-level and brain-region information when EEG is reduced to a single vector, loss of temporal dynamics when video is similarly summarized, and the lack of a mechanism for integrating global pretrained representations with these finer structural features. EVER addresses these limitations through the following components:

  • Global EEG representation: A pretrained EEG model, CBraMod, encodes the entire trial into a single vector.
  • Brain-region representations (BAR-GCN): Each electrode signal is transformed into a time–frequency representation using STFT and encoded into a vector. Electrodes are grouped into frontal, temporal, central, parietal, and occipital regions based on scalp location. A GCN exchanges information only among electrodes within the same region, followed by region-level aggregation. The authors describe this design as montage-flexible, since datasets with different electrode layouts can be mapped to the same anatomical regions.
  • Video temporal structure (VCG): Facial videos are divided into overlapping 32-frame clips at approximately two-second intervals. A pretrained VideoMAE or AdaMAE model extracts clip-level features, which are augmented with positional information. Connections between clips follow a Gaussian function of temporal distance, with a learnable width parameter σ.
  • Inter-modal graph: Video clip nodes, the global EEG node, and regional EEG nodes are integrated into a graph with predefined permissible connections: clip–clip, clip–global EEG, clip–region, global EEG–region, and region–region. Final predictions combine the video and EEG branch outputs using learned, class-specific fusion weights ω that are shared across samples.

Training combines class-weighted cross-entropy and focal loss to address class imbalance.

Experiments are conducted on Emognition, with four EEG channels and 319 training and 88 evaluation trials, including eight evaluation subjects, and MDMER, with 18 EEG channels and approximately 1.8K training and 0.5K evaluation trials, including 15 evaluation subjects. Comparisons include five video-only methods, five EEG-only methods, and two EEG–video fusion methods, EAV and Milmer. Performance is measured using accuracy, unweighted average recall (UAR), and weighted F1, averaged across valence and arousal. Results are reported on a single subject-independent split, without reporting repeated runs across random seeds.

EVER achieves the highest reported average accuracy on both datasets: 38.6% on Emognition, compared with 35.3% for the next-best method, CBraMod, and 35.5% on MDMER, compared with 33.4% for VideoMAE. Additional analyses include component ablations, robustness to EEG noise, bootstrap confidence intervals, modality complementarity visualized through Venn diagrams, and comparisons with six additional EEG models.

Critical strengths
  1. A useful benchmark under a unified evaluation protocol.
    The authors evaluate 12 baseline models on two public datasets with synchronized EEG and facial video, Emognition and MDMER: five video-only models, five EEG-only models, and two multimodal models. All models are evaluated using subject-independent splits, common loss functions, and a consistent training budget. In a field where paired datasets are small and standardized comparisons are limited, evaluating representative video and EEG models under a unified protocol provides a valuable reference for future research. The use of public datasets and publicly available pretrained models also facilitates reproduction and extension.

Notably, the benchmark shows that the stronger modality varies by dataset. On Emognition, EEG models achieve higher average accuracy (27.6–35.3%) than video models (23.9–30.7%), whereas the opposite holds on MDMER, where video models (29.5–33.4%) outperform EEG models (25.1–30.2%). This reversal suggests that neither modality consistently dominates across datasets and provides empirical support for the paper’s motivation to combine EEG and video.

  1. A coherent framework for integrating global representations with spatial and temporal structure.
    Unlike conventional approaches that summarize each modality into a single vector before fusion (Section 2.1), EVER retains the global representation from a pretrained EEG model while jointly modeling regional EEG nodes and temporally ordered video clip nodes within a single graph. This design directly addresses the identified loss of brain-region information and facial dynamics during global aggregation.

Moreover, existing region-based EEG graphs are tied to particular electrode layouts and require redefinition when the dataset changes (Section 2.2). EVER instead maps electrodes to standard regions defined by the 10–20 system, allowing the same graph structure to accommodate datasets with different electrode configurations without redesign (Section 4.1.1). Given the variation in electrode layouts across EEG devices, this provides a reasonable and broadly applicable framework for incorporating structural prior knowledge while retaining the benefits of pretrained backbones.

Critical weaknesses
  1. The reliability of the reported results is difficult to assess because of numerical inconsistencies and unclear evaluation procedures.**

(a) EEG-only models and EVER may have been evaluated on different sample sets. Emognition contains 88 evaluation samples (Section 4.1.1 and Appendix A.3), so each accuracy should be a rounded value of an integer divided by 88. Of the 18 accuracy values reported for the video models, EAV, Milmer, and EVER, 17 satisfy this condition exactly—for example, EVER’s arousal accuracy of 0.409 corresponds to 36/88. However, several EEG-only results cannot arise from 88 samples. For instance, ST-Transformer’s arousal accuracy of 0.400 cannot be obtained even through rounding: the nearest possible values are 0.398 (35/88) and 0.409 (36/88). It does, however, correspond exactly to 34/85. Similarly, ST-Transformer’s valence accuracy of 0.235 (20/85), LaBraM’s valence accuracy of 0.259 (22/85), and REVE’s arousal accuracy of 0.259 (22/85) are incompatible with 88 samples but consistent with 85. Among sample counts from 80 to 90, only 85 satisfies all these values. The results for the six additional EEG models in Appendix A.6 are also consistent with 85 samples.

The paper does not report excluding samples from the EEG-only evaluation. Consequently, the central comparison—EVER at 38.6% versus CBraMod at 35.3%—may involve different evaluation sets. This would also contradict the statement in Appendix A.3 that all models were evaluated using the same resampled observations.

(b) The reported gains amount to only a few predictions, and repeated runs are not reported. On Emognition, a one-percentage-point increase in accuracy averaged across valence and arousal corresponds to approximately two correct predictions. The headline improvement of 3.3 percentage points therefore represents approximately six additional correct predictions across the two tasks. These results come from a single split, without reported repetitions across random seeds. The bootstrap intervals in Appendix A.3 resample evaluation observations while holding a trained model fixed, so they do not capture variability due to retraining. Even these intervals overlap substantially: [.312, .460] for EVER and [.279, .427] for CBraMod.

(c) Early stopping is described without an identified validation set. Appendix A.7.1 states that training stops when the “validation score” does not improve for 15 epochs, but Section 4.1.1 describes only training and evaluation splits. If the evaluation set was used to select the stopping point, the results in Tables 1–5, including the small differences discussed above, could reflect selection on the evaluation data.

(d) Class-balanced performance on MDMER is close to chance. For a five-class task, uniform random prediction or constant prediction of a single class yields a UAR of 0.20. Across all models, average UAR on MDMER ranges from 0.185 to 0.238, with ST-Transformer (0.185) and LaBraM (0.195) falling below 0.20. The paper acknowledges that MDMER labels are concentrated around the middle scores (Appendix A.7.2), so a majority-class baseline could achieve substantial accuracy, yet no such baseline is reported. For example, VideoMAE’s MDMER arousal results—accuracy 0.313, UAR 0.211, and weighted F1 0.178—resemble the pattern expected from a majority-class predictor: accuracy equal to the majority-class proportion, UAR of 0.20, and weighted F1 roughly half the accuracy. It therefore remains unclear whether the accuracy differences on MDMER reflect improved discrimination among emotions.

(e) The appendix analyses are not numerically consistent with the main results. Fig. A1 and Appendix A.4 do not identify whether “Video” and “Ours” use VideoMAE or AdaMAE. The MDMER evaluation set is described only as approximately 0.5K samples. Estimating its size as approximately 475 from the percentages in Fig. A1, the EEG model makes 30 + 34 + 7 + 81 = 152 correct predictions, corresponding to 0.320 and matching CBraMod in Table 1. However, the video model makes 5 + 34 + 43 + 81 = 163 correct predictions, corresponding to 0.343. This matches neither VideoMAE’s valence accuracy of 0.355 (approximately 169 correct predictions) nor AdaMAE’s 0.359 (approximately 170). The predictions used for the complementarity analysis may therefore come from a different run than those reported in the main table. Additionally, Table A1 reports identical values for all three metrics for AdaMAE-based EVER at noise levels λ = 0.1 and 0.2: 0.365/0.297/0.292.

  1. The central claims rely on single-run ablation results without supporting qualitative or interpretive evidence.

The central contribution is that preserving brain-region structure and clip-level temporal structure improves multimodal fusion. This claim rests on the ablation results in Tables 2–4, but the differences on Emognition are small, as discussed above. For example, in Table 2, adding both clip nodes and VCG improves accuracy by only 1.1 percentage points over the BAR-GCN-only configuration (0.375), equivalent to approximately two additional correct predictions across the two tasks. The results are also sensitive to the configuration: removing either positional information or the Gaussian kernel from VCG reduces the multimodal model’s accuracy to 0.307 or 0.290, respectively (Table 4), below the EEG-only result of 0.353 despite retaining an EEG branch. No further evidence establishes that these numerical changes arise from the mechanisms claimed by the authors.

  • Qualitative analysis is absent. The figures consist of an architecture diagram (Fig. 1), Venn diagrams of correct predictions (Fig. A1), label distributions (Fig. A2), and example input frames (Figs. A3 and A4). None explains what the model uses to make its predictions. Electrode attention within regions, α (Eq. 6), the learned temporal width, σ (Eq. 8), and class-specific fusion weights, ω (Eq. 13), are already available within the model but are not reported. Although Fig. 1 illustrates local versus global behavior for different values of σ, the learned values are not provided. In particular, Emognition’s four EEG channels are assigned to two regions, leaving only one to three nodes in each regional graph. Without interpretive analysis, it is unclear what BAR-GCN learns in this setting. Confusion matrices could also help resolve the concern about class-concentrated predictions raised in point 1(d), but none are provided.

  • The premise underlying VCG is not substantiated. Section 3.3 motivates the Gaussian temporal kernel by assuming that emotional cues in video are temporally local, such that nearby clips are more likely to share the same emotional state. No analysis directly supports this assumption. Appendix A.7.2 presents Figs. A3 and A4 as illustrations of this motivation, but each figure shows only the 32 frames within a single clip, rather than relationships between clips. Visually, both examples show little change in expression beyond blinking, including the disgust-stimulus example in Fig. A3. Fig. A4 also lacks an emotion label or stimulus description. Thus, the quantitative results in Table 4 are the only evidence offered that meaningful facial changes occur across clips and that VCG captures them.

  • The validation scope is narrow. Each ablation is conducted on only one dataset, either Emognition or MDMER, using AdaMAE. No ablations are provided for VideoMAE, even though that combination achieves better UAR and weighted F1 on both datasets. The comparison with alternative fusion strategies using the same backbone (Table 5) is likewise restricted to Emognition with AdaMAE. Furthermore, every experiment uses CBraMod as the EEG backbone, leaving the applicability of the proposed “unified framework” to other EEG backbones untested.

  • The complementarity claim in Appendix A.4 is not established by Fig. A1. This analysis is presented as evidence for the abstract’s claim that the model recognizes emotional patterns unavailable from either modality alone and for the discussion of cross-modal complementarity in Section 4.2. The percentages in Fig. A1 suggest an MDMER evaluation set of approximately 475 samples; modest variation in this estimate would not change the following observations.

    (a) Overlap between the unimodal models’ correct predictions. For arousal, the video model correctly predicts 20 + 1 + 104 + 23 = 148 samples (pv≈0.31p_v \approx 0.31), while the EEG model correctly predicts 98 + 1 + 13 + 23 = 135 (pe≈0.28p_e \approx 0.28). If their correctness were independent, the expected number of jointly correct predictions would be approximately 475×0.31×0.28≈42475 \times 0.31 \times 0.28 \approx 42, whereas the observed overlap is only 1 + 23 = 24. Confusion matrices are needed to distinguish whether their different correct predictions reflect complementary information or a tendency to concentrate predictions on different classes. This pattern is specific to arousal: for valence, the observed overlap is 34 + 81 = 115, substantially greater than the independence-based expectation of approximately 52.

    (b) Samples correctly classified only by EVER. The authors present samples that both unimodal models misclassify but EVER classifies correctly as evidence of gains beyond simply selecting the better modality. For arousal, however, both unimodal models are wrong on 475 − (20 + 1 + 104 + 23 + 98 + 13) = 216 samples, of which EVER correctly classifies only 17, or approximately 7.9%. For valence, EVER correctly classifies 48 of the 475 − (5 + 34 + 43 + 81 + 30 + 7) = 275 jointly misclassified samples, or approximately 17.5%. Both rates are below the 20% expected from uniform random guessing in a five-class task. The existence of these corrected samples alone therefore provides limited evidence that fusion has captured additional information unavailable to the unimodal models.

Questions for authors
  1. Evaluation sample counts: As noted in Weakness (a), several EEG-only accuracies on Emognition are consistent with 85 evaluation samples rather than 88—for example, ST-Transformer’s arousal accuracy of 0.400 = 34/85. Were the EEG-only models evaluated on a different sample set? If so, please identify the excluded samples and explain why they were excluded. Please also provide comparisons between EVER and the EEG-only models on the same sample set, including updated bootstrap intervals corresponding to Appendix A.3.

  2. Validation and repeated runs: On which data was the “validation score” used for early stopping in Appendix A.7.1 computed? Using a validation set separate from the evaluation set, please report means and standard deviations across at least 3–5 random seeds, preferably also across multiple subject splits. In particular, it is necessary to establish whether EVER consistently outperforms the EEG-only CBraMod baseline across repeated runs.

  3. Baselines and prediction distributions: For both datasets, please provide the five-class label distributions for the training and evaluation sets separately, the accuracy and UAR of a majority-class predictor, and confusion matrices for the main models. Is EVER’s accuracy of 35.5% on MDMER meaningfully higher than the majority-class baseline?

  4. Isolating the contribution of the fusion architecture: The comparison of fusion strategies with matched backbones in Table 5 is limited to Emognition with AdaMAE + CBraMod. Please extend these comparisons, including concatenation and the other strategies in Table 5, as well as prediction averaging, to both datasets and both backbone combinations.

  5. Complementarity analysis (Appendix A.4):

    (a) Which backbone in Table 1—VideoMAE or AdaMAE—and which experimental run produced “Video” and “Ours (Video + EEG)” in Fig. A1? As noted in Weakness (e), the number of correct video predictions matches neither backbone’s reported result.

    (b) As calculated in the weaknesses, EVER’s accuracy on samples misclassified by both unimodal models is below the uniform-random expectation of 20%. What supports interpreting these corrected predictions as evidence of information gained through fusion? Please provide the same analysis using models trained on the same modality with different random seeds as a control.

  6. Motivation and operation of VCG:

    (a) Section 3.3 motivates VCG through the assumption that temporally adjacent clips are more likely to share the same emotional state. However, Appendix A.7.2 refers to Figs. A3 and A4, which illustrate changes within individual clips. Is VCG intended to capture changes within clips or between clips?

    (b) Please show representative frames from successive clips across an entire trial, together with the corresponding emotion label, to demonstrate meaningful changes in facial expression between clips.

    (c) Please report the final learned values of σ, the distribution of the number of clips per sample, NN, and examples of inter-clip affinity matrices. The initial σ of 3.0 corresponds to approximately three clips, or six seconds.

  7. Qualitative results and interpretation: Please report the class-specific fusion weights ω and the distributions of within-region attention weights α. Please also specify how Emognition’s four electrodes are assigned to the frontal and temporal regions and illustrate what BAR-GCN learns when each regional graph contains only one to three nodes.

  8. EEG length handling: Section 3.1.2 states that each EEG sequence is resized to 2,000 samples, while each example spans an entire stimulus trial and the number of video clips varies with its duration (Sections 3.1.1 and 4.1.1). This appears to imply that the number of EEG samples representing one second varies across trials. Please report the distribution of trial durations and explain how this processing is compatible with pretrained CBraMod’s input representation, which uses 200-sample patches.

Suggestions for improvement

  1. Report results across multiple subject splits and random seeds, with significance tests. The authors acknowledge in Appendix A.3 that the current differences on Emognition are not statistically distinguishable.

  2. Extend the ablations to both datasets and both backbone combinations.

  3. Evaluate generality across backbones. All experiments use CBraMod as the EEG backbone. Applying both EVER and a simple fusion baseline to representative alternative combinations from Table 1, such as Swin + BIOT and VideoMAE + LaBraM, would help establish whether the proposed architecture remains effective across backbone choices.

  4. Analyze overfitting. The projection in Eq. (3) contains approximately 25.2M parameters, accounting for most of the approximately 28M additional parameters reported in Appendix A.2. Given that Emognition contains only 319 training samples, reporting training–evaluation performance gaps or learning curves would strengthen confidence in the results.

  5. Correct the citations. The citation supporting “Video captures … facial expressions” in Section 1, Zhang et al. (2020), concerns multimodal physiological-signal fusion. Herwig (2003, TMS localization) and Macorig (2021, prevalence of benign EEG variants), cited in Section 2.2 to support standard region definitions under the 10–20 system, are application studies using that system rather than appropriate references for the region definitions. The reference for “Video Swin Transformer (Liu et al., 2021)” in Section 2.1 is the image-based Swin paper from ICCV 2021; Video Swin was published at CVPR 2022. In Section 2.2, the name “MS-GCN (Du et al., 2022)” also does not match the cited title, which uses “multi-dimensional.”

  6. Correct the mathematical notation. Malformed subscripts around Eqs. (5) and (6), including “w * a,” “z * rk,” and “GCN∗1,” should be corrected.

4 comments
AuthorsOpenReview ↗

Response to Reviewer k3Z3 (1/4) - Evaluation sample counts (W1(a), Q1)

We thank the reviewer for the thorough and careful reading and for recognizing the value of the unified benchmark and the montage-flexible design. We refer to the two numbered weaknesses as W1 and W2, to the four bullet points under W2 as W2.1 to W2.4, and to the questions as Q1 to Q8. In brief, we add (i) a re-evaluation of the EEG-only models and EVER on identical test trials, which resolves the sample-count discrepancy, (ii) results without checkpoint selection and repeated runs over three seeds, (iii) majority-class, label-distribution and prediction-spread analyses, (iv) ablations on the second dataset and fusion comparisons on both datasets and both video backbones, (v) the learned VCG and gate parameters, and (vi) answers on the operation of VCG and on the EEG preprocessing.

W1(a) and Q1. Evaluation sample counts

We apologize for the confusion caused by the inconsistent sample counts. The reviewer is correct, and no sample was excluded. The difference comes from distributed evaluation. The EEG-only models were evaluated on one GPU over the 85 unique Emognition test trials. The multi-GPU runs use a distributed sampler that pads the test set to a multiple of the number of GPUs, which duplicates 3 trials and yields 88. On MDMER, the same padding adds a single trial to the 475 unique trials. We re-evaluated all models on the unique trials, so that every interval and paired test is computed on identical observations, as stated in Appendix A.3. The results are shown in Tab. R1. The ranking of the models in accuracy is unchanged, and the revision will report every model on the unique trials. The results in this response follow the same protocol.

Table R1. Results on the Emognition dataset over the 85 unique test trials (95% bootstrap CI, B = 10,000).

ModelAccUARW-F1
ST-Transformer0.318 [0.253, 0.382]0.267 [0.214, 0.323]0.259 [0.195, 0.328]
BIOT0.324 [0.247, 0.400]0.285 [0.217, 0.356]0.306 [0.228, 0.387]
LaBraM0.276 [0.200, 0.359]0.237 [0.208, 0.264]0.139 [0.085, 0.210]
CBraMod0.353 [0.276, 0.429]0.287 [0.229, 0.347]0.292 [0.216, 0.369]
REVE0.276 [0.206, 0.347]0.238 [0.183, 0.297]0.240 [0.172, 0.312]
EVER (AdaMAE + CBraMod)0.371 [0.294, 0.447]0.306 [0.252, 0.360]0.298 [0.223, 0.377]
EVER (VideoMAE + CBraMod)0.371 [0.294, 0.447]0.353 [0.272, 0.435]0.353 [0.273, 0.433]

On the identical trials, EVER with VideoMAE improves over CBraMod by 6.6 points in UAR and by 6.1 points in W-F1. The intervals in Tab. R1 are computed for each model separately, so their overlap does not show whether the difference between two models is significant. We therefore test directly whether the gain of EVER over the strongest EEG model, and over a simple combination of the same backbones, exceeds the sampling variability of the test set. We use MDMER for this test, where the test set is five times larger. We resample the 475 test trials with replacement 10,000 times and compute the difference between EVER and each compared model on the same resampled trials. The two-sided p-value is twice the fraction of resamples in which the sign of the difference is reversed. As shown in Tab. R2, EVER is significantly better than CBraMod and the logit-averaging ensemble of the same backbones in all three metrics.

Table R2. Paired comparison on the MDMER dataset over the 475 unique test trials (two-sided paired-bootstrap p-value of the difference from EVER in parentheses).

ModelAcc (p-value)UAR (p-value)W-F1 (p-value)
EVER (VideoMAE + CBraMod)0.3540.2370.280
CBraMod (best EEG)0.302 (0.012)0.208 (0.007)0.219 (0.001)
Logit-averaging ensemble of CBraMod and VideoMAE0.306 (0.019)0.208 (0.011)0.217 (0.001)
AuthorsOpenReview ↗

Response to Reviewer k3Z3 (2/4) - Validation, repeated runs and baselines (W1(b), W1(c), W1(d), Q2, Q3)

W1(b), W1(c), Q2 and Suggestion 1. Validation and repeated runs

Regarding model selection, the validation score in Appendix A.7.1 is computed on the evaluation split. Every model is reported at its best epoch under this identical protocol, for all 10 unimodal backbones and all fusion models, so any optimism of the selection applies to every model alike. Holding out validation subjects would further shrink the small training sets (319 trials in Emognition). The protocol is needed because, in the subject-independent setting, even large pretrained backbones overfit the training subjects within a few epochs, and the evaluation loss rises thereafter. A fixed final epoch would compare overfitted models rather than the capacity of each method.

This choice does not create the advantage of EVER. As shown in Tab. R3, no compared model exceeds EVER in any metric on either dataset without checkpoint selection, at the final epoch or over the last five epochs. To show both backbone combinations directly, EVER uses AdaMAE + CBraMod on Emognition and VideoMAE + CBraMod on MDMER. At these epochs the accuracies of most models fall below the majority-class level, which confirms this overfitting. Relative to the last-five mean, the selection raises the accuracy of CBraMod by 0.112 on Emognition and 0.062 on MDMER, and that of EVER by 0.084 and 0.067, so it does not favor EVER.

Table R3. Results without checkpoint selection (Acc / UAR / W-F1, from the training logs).

ModelEmognition dataset, final epochEmognition dataset, last-5 meanMDMER dataset, final epochMDMER dataset, last-5 mean
CBraMod (EEG)0.229 / 0.228 / 0.2340.241 / 0.225 / 0.2320.261 / 0.218 / 0.2590.240 / 0.205 / 0.245
VideoMAE (video)0.256 / 0.213 / 0.2160.266 / 0.227 / 0.2330.265 / 0.225 / 0.2680.284 / 0.227 / 0.273
AdaMAE (video)0.244 / 0.225 / 0.2380.255 / 0.234 / 0.2440.275 / 0.215 / 0.2560.270 / 0.202 / 0.257
EVER0.330 / 0.300 / 0.2680.287 / 0.253 / 0.2440.287 / 0.235 / 0.2780.287 / 0.236 / 0.279

Regarding repeated runs, our compute within the rebuttal period limits them to EVER and CBraMod, which is the EEG-only counterpart of EVER and is light enough to retrain repeatedly. The repeated runs use the same two combinations. The results over three seeds are shown in Tab. R4. EVER is higher on all three metrics on both datasets. The gain is clearest on UAR and W-F1, the two metrics that account for class imbalance. UAR rises from 0.283 to 0.310 on Emognition and from 0.209 to 0.225 on MDMER. W-F1 rises from 0.272 to 0.301 on Emognition and from 0.234 to 0.269 on MDMER, a relative improvement of 11% and 15%. This indicates that EVER improves the recognition across the emotion classes, not only that of the frequent ones. Acc shows some variation across runs, and its mean remains higher for EVER. The advantage of EVER is not specific to the single run reported in the main paper, and it holds on average over repeated runs on both datasets despite their small test sets.

Table R4. Mean ± standard deviation over three seeds (Acc / UAR / W-F1).

ModelEmognition datasetMDMER dataset
CBraMod0.322 ± 0.030 / 0.283 ± 0.036 / 0.272 ± 0.0330.295 ± 0.009 / 0.209 ± 0.003 / 0.234 ± 0.014
EVER0.341 ± 0.026 / 0.310 ± 0.013 / 0.301 ± 0.0080.321 ± 0.029 / 0.225 ± 0.014 / 0.269 ± 0.021

Multiple subject splits require a full training run per split for every model, which is not feasible within the rebuttal period, and we will discuss this as a limitation.

W1(d) and Q3. Baselines and prediction distributions

Tab. R5 reports the five-class label distributions of the training and test sets and the majority-class accuracy of each target. The majority-class predictor outputs the most frequent test class of each target. Its accuracy equals the share pp of that class, its UAR is 0.20, and its W-F1 is 2p2/(1+p)2p^2/(1+p). Averaged over valence and arousal, the predictor reaches 0.300 / 0.200 / 0.139 in Acc / UAR / W-F1 on Emognition and 0.317 / 0.200 / 0.153 on MDMER. EVER reaches 0.371 / 0.353 / 0.353 on Emognition, as shown in Tab. R1, and 0.354 / 0.237 / 0.280 on MDMER. EVER therefore exceeds this predictor in all three metrics on both datasets.

Table R5. Five-class label counts (classes 0 to 4) and majority-class accuracy on the test set.

Dataset, targetTraining setTest set (unique trials)Majority-class Acc (test)
Emognition, valence38 / 67 / 72 / 106 / 3611 / 24 / 17 / 22 / 110.282
Emognition, arousal90 / 96 / 43 / 70 / 2017 / 19 / 14 / 27 / 80.318
MDMER, valence90 / 302 / 367 / 869 / 21785 / 136 / 74 / 159 / 210.335
MDMER, arousal300 / 652 / 378 / 430 / 8563 / 142 / 134 / 125 / 110.299
AuthorsOpenReview ↗

Response to Reviewer k3Z3 (3/4) - Appendix consistency, ablations and VCG (W1(e), W2.1, W2.2, Q5(a), Q6, Q7)

W1(d) and Q3 (continued)

VideoMAE indeed concentrates almost all of its MDMER arousal predictions on the majority class, as the reviewer inferred from its metrics. This behavior does not come from a weak or poorly trained baseline. VideoMAE is a widely used self-supervised video foundation model, fine-tuned here in the same way as every video backbone of the benchmark, with the final encoder block and the classification head updated (Section 3.1.1). The result instead reflects the difficulty of the task. In the subject-independent setting, facial responses to the stimuli are subtle and vary across subjects, so video alone gives little evidence to separate the five arousal levels, and the model falls back to the dominant class. This limitation motivates an EEG–video framework. With the same VideoMAE backbone, EVER predicts four to five distinct classes for each target on both datasets, which shows that the structured EEG pathway restores discrimination among the emotion classes. We will add the confusion matrices of the main models to the appendix.

W1(e) and Q5(a). Consistency of the appendix

We thank the reviewer for checking these numbers. The predictions in Fig. A1 were taken from a different evaluation run of the unimodal baselines than the one reported in Tab. 1 of the main paper, which explains the mismatch in the number of correct video predictions. In the revision, we will regenerate Fig. A1 so that it uses the same run as Tab. 1 of the main paper, state that the figure uses the VideoMAE-based models, and give the exact size of the MDMER test set (475 trials). The duplicated entry for λ = 0.1 and 0.2 in Tab. A1 of the appendix is incorrect and will be corrected.

W2, W2.1 and Q7. Ablations and learned parameters

We first address the robustness of the ablations. They now cover a second dataset, as shown in Tab. R6. On MDMER, adding clip nodes without temporal structure to BAR-GCN leaves the model near the majority-class level, and W-F1 falls from 0.220 to 0.171. Only the full VCG raises all three metrics, to 0.341 / 0.231 / 0.274, and removing either the positional encoding or the Gaussian prior lowers all three. The full configuration therefore performs best on a test set five times larger than Emognition. On Emognition, the single-component removals fall below CBraMod because EEG is much stronger than video there. Removing the temporal structure of the clip nodes reduces VCG to an unstructured fusion of many clip features, which dilutes the EEG evidence.

Table R6. Ablations on the MDMER dataset with AdaMAE + CBraMod over the 475 unique test trials (Acc / UAR / W-F1).

ConfigurationAcc / UAR / W-F1
BAR-GCN only0.281 / 0.188 / 0.220
+ Clip nodes0.323 / 0.205 / 0.171
+ Clip nodes + VCG, w/o positional encoding0.308 / 0.197 / 0.186
+ Clip nodes + VCG, w/o Gaussian prior0.319 / 0.202 / 0.165
Full EVER0.341 / 0.231 / 0.274

Regarding the interpretive parameters, the learned temporal width σ lies between 2 and 4 clips on both datasets, which corresponds to a local window of roughly 4 to 8 s. The class-specific gate ω converges to about 0.27 during training. Regarding the region assignment, the electrodes of both montages are mapped to the canonical regions of the 10–20 system (Section 4.1.1), and only the regions that contain available electrodes are instantiated. This yields five region nodes for the 18-channel MDMER montage and two region nodes for the 4-channel Emognition montage, each with two of the four Muse electrodes (AF7, AF8, TP9 and TP10). With two electrodes per region, BAR-GCN reduces to a pairwise interaction followed by attention pooling, and its role is to supply region nodes that interact with the clip nodes in the inter-modal graph. The benefit of the anatomical grouping itself is demonstrated on MDMER with 18 channels, where random partitions perform worse than anatomical ones, as shown in Tab. 3 of the main paper. We will report the distribution of the within-region attention weights α in the appendix.

W2.2 and Q6. Motivation and operation of VCG

VCG models relationships between clips, not within clips. The Gaussian prior weights the edge between two clips by their temporal distance. Figs. A3 and A4 show the frames within a single clip, and we will replace them with frames from successive clips across a whole trial together with the emotion label. The number of clips per test trial is 56.7 on average on Emognition (from 10 to 149) and 12.4 on MDMER (from 9 to 15), so every trial contains many temporally ordered clips. At σ = 3, which lies within the learned range, the affinity between two clips stays above 0.6 within three clips and falls below 0.14 beyond six clips, so each clip mainly exchanges information with its temporal neighborhood. Since OpenReview does not allow figures, we will add these frames and example affinity matrices to the appendix.

AuthorsOpenReview ↗

Response to Reviewer k3Z3 (4/4) - Fusion comparison, complementarity and EEG input (W2.3, W2.4, Q4, Q5(b), Q8)

W2.3, Q4 and Suggestions 2 and 3. Isolating the contribution of the fusion architecture

The fusion comparison on the Emognition dataset with identical backbones is already reported in Tab. 5 of the main paper. We extended this comparison to the MDMER dataset with both video backbones, as shown in Tab. R7, and added prediction averaging for both backbone combinations on both datasets, as shown in Tab. R8. With identical backbones, loss and hyperparameters, concatenation, cross-attention and FiLM collapse to the majority-class predictor on MDMER for both video backbones, whereas EVER does not. EVER also outperforms the logit-averaging ensemble of the same backbones for both combinations on both datasets in all three metrics. The comparison on Emognition with VideoMAE will be added in the revision. Regarding other EEG backbones, EVER also improves over the corresponding EEG-only model in all three metrics when CBraMod is replaced by REVE or by LaBraM, including the VideoMAE + LaBraM combination suggested by the reviewer, as shown in Tab. R9.

Table R7. Fusion strategies with identical backbones on the MDMER dataset.

Fusion (Acc / UAR / W-F1)AdaMAE + CBraModVideoMAE + CBraMod
Majority-class predictor0.317 / 0.200 / 0.1530.317 / 0.200 / 0.153
Concatenation0.317 / 0.200 / 0.1530.317 / 0.200 / 0.153
Cross-attention0.317 / 0.200 / 0.1530.317 / 0.200 / 0.153
FiLM0.317 / 0.200 / 0.1530.317 / 0.200 / 0.153
Pooled-node GCN0.320 / 0.202 / 0.1670.296 / 0.195 / 0.156
EVER (ours)0.341 / 0.231 / 0.2740.354 / 0.237 / 0.280

Table R8. Logit-averaging ensemble versus EVER with the same backbones (Acc / UAR / W-F1, over the unique test trials of each dataset).

BackbonesModelEmognition datasetMDMER dataset
CBraMod + AdaMAEEnsemble0.353 / 0.288 / 0.2940.300 / 0.203 / 0.212
CBraMod + AdaMAEEVER0.371 / 0.306 / 0.2980.341 / 0.231 / 0.274
CBraMod + VideoMAEEnsemble0.359 / 0.292 / 0.2980.306 / 0.208 / 0.217
CBraMod + VideoMAEEVER0.371 / 0.353 / 0.3530.354 / 0.237 / 0.280

Table R9. EVER with other EEG backbones versus the corresponding EEG-only model (Acc / UAR / W-F1).

BackbonesDatasetEEG-onlyEVER
AdaMAE + REVEEmognition0.276 / 0.238 / 0.2400.347 / 0.267 / 0.260
VideoMAE + LaBraMEmognition0.276 / 0.237 / 0.1390.324 / 0.259 / 0.281

W2.4 and Q5(b). Complementarity analysis

The reviewer's reading is correct. On MDMER arousal, the video model and CBraMod each concentrate their predictions on a different class, so their correct predictions rarely overlap, whereas for valence both favor the same class, which produces the large overlap. The overlap statistics in Fig. A1 therefore reflect class concentration rather than complementary information. We agree that the samples corrected only by EVER are not by themselves sufficient evidence of information gained through fusion. We will revise the complementarity claim in Appendix A.4 accordingly and add the same-modality, different-seed control suggested by the reviewer, using the repeated runs described under W1(b).

Q8. EEG length handling

Trial durations range from 11.5 to 321.8 s on Emognition (median 119.7 s) and from 20 to 33 s on MDMER (median 29.0 s). Since each label describes the whole stimulus interval, the global branch resamples the complete trial to a fixed input of ten 200-sample patches. As the reviewer notes, the time span of a patch varies across trials. The input keeps the 200-sample patch format of CBraMod, which is initialized from the pretrained weights and fully fine-tuned, so it adapts to this time scale. The EEG-only CBraMod baseline uses the same input. This branch captures the global, slowly varying dynamics. The fine spectral content is preserved by the parallel BAR-GCN branch, which computes the STFT of each channel on the raw signal at its native sampling rate. We will describe this design in the revision.

Suggestions 4 to 6. Overfitting, citations and notation

We apologize for the errors in the citations and the notation, and we thank the reviewer for identifying them. We will add the learning curves as an analysis of overfitting. They show that the training loss keeps decreasing while the evaluation loss rises after a few epochs. We will also correct all citations the reviewer pointed out. Zhang et al. (2020) will be replaced with a reference on facial expression, Herwig (2003) and Macorig (2021) with a reference that defines the 10–20 regions, Video Swin will be cited as Liu et al., CVPR 2022, and the name of Du et al. (2022) will follow its title. The malformed subscripts around Eqs. (5) and (6) will be fixed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.