- Rating
- 1
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
EVER addresses EEG–video emotion recognition. It argues that existing fusion methods pool each modality into one vector, discarding the anatomical organization of EEG channels and the temporal evolution of facial behavior. EVER keeps these fine-grained structures and fuses them with pretrained global representations in one graph.
Contributions.
BAR-GCN: per-channel EEG spectrogram embeddings are connected by a GCN only within 10–20 scalp regions, then pooled into region nodes. Mapping to standard regions lets different electrode montages share the scheme. VCG: video clip embeddings from VideoMAE or AdaMAE become temporally ordered nodes, with sinusoidal position codes and a learnable Gaussian that weakens edges between distant clips. Inter-modal graph: connects the clip nodes, the global CBraMod EEG embedding and the region nodes under a structured edge mask, followed by a 2-layer GCN and a per-class gate between the EEG and video outputs. Benchmark: 12 unimodal and multimodal models compared on Emognition and MDMER under subject-independent splits.
Validation.
Main results: 5-class valence and arousal classification. EVER reaches 38.6% / 35.5% average accuracy, against 35.3% (CBraMod) and 33.4% (VideoMAE) for the best baselines. Ablations: components, anatomical vs. random region partitions, VCG parts and fusion operators. Appendix analyses: bootstrap confidence intervals, EEG-noise robustness, model size, modality complementarity and additional EEG encoders.
Critical strengths
A sensible fusion design for an underexplored pairing. Rather than pooling each modality, EVER keeps EEG regions and ordered video clips as separate nodes and lets them interact with the pretrained global embedding. Mapping electrodes to canonical scalp regions also lets the same model handle different montages (4 and 18 channels here), which matters for a field where every dataset uses a different headset. More statistical candor than is typical for the area. All models share one subject-independent split, loss and training budget. The authors report bootstrap confidence intervals and explicitly say the differences on Emognition are not statistically separated. That honesty makes the paper easier to assess, even though it also exposes how thin the evidence is.
Critical weaknesses
The evaluation can’t distinguish the gains from noise. Emognition’s test set is 88 samples from 8 subjects, and there is a single subject split and a single training run. EVER’s +3.3-point lead over CBraMod amounts to about three samples per target, inside the paper’s own confidence intervals. Early stopping uses a “validation score,” but only train and test splits are described. If the test set was used for model selection, every number is optimistically biased. No majority-class baseline is given for the 5-way binned labels, so it’s unclear how far above trivial the 24–39% accuracies are. For a result at this scale to be convincing, it needs leave-one-subject-out or repeated subject splits with variance across seeds. The ablations and baselines behave in ways that undermine the claimed mechanism. Removing only the positional encoding or only the Gaussian kernel drops EVER to 30.7% and 29.0% (Table 4), below the EEG-only CBraMod model (35.3%) that sits inside EVER. All four fusion baselines in Table 5 (concatenation, cross-attention, FiLM, pooled-node GCN) also fall below EEG alone. Fusion that makes a model worse than its strongest input points to unstable training or poorly tuned baselines, not to the graph structure being essential. The complementarity analysis also shows the fused model mostly follows video: on MDMER arousal, it keeps only 13 of the 111 samples that EEG alone gets right. A simple logit-averaging ensemble of the same two backbones is missing, and it’s the baseline most likely to show whether the graph machinery is needed at all.
Questions for authors
Which set is used for early stopping? If it’s not the test set, what are its size and subjects? Can you report leave-one-subject-out or repeated subject-split results (mean ± std) over several seeds? What are the majority-class and chance accuracies per target after 5-way binning? How does a logit-averaging ensemble of the same CBraMod and VideoMAE backbones perform? Why do all fusion baselines in Table 5, and the single-component removals in Table 4, fall below the EEG-only model?
8 comments
Response to Reviewer qnJ1 (2/3) - Repeated runs, majority baseline and ensemble (Q2, Q3, Q4)
Q2. Leave-one-subject-out or repeated runs over seeds
Although the evaluation with held-out validation subjects under Q1 is already reported over three seeds, we also report repeated runs of EVER and CBraMod under the original protocol of the main paper. To show both backbone combinations of EVER effectively, EVER uses AdaMAE + CBraMod on Emognition and VideoMAE + CBraMod on MDMER. The results over three seeds are shown in Tab. R2. EVER is higher on all three metrics on both datasets. The gain is clearest on UAR and W-F1, the two metrics that account for class imbalance. UAR rises from 0.283 to 0.310 on Emognition and from 0.209 to 0.225 on MDMER. W-F1 rises from 0.272 to 0.301 on Emognition and from 0.234 to 0.269 on MDMER, a relative improvement of 11% and 15%. This indicates that EVER improves the recognition across the emotion classes, not only that of the frequent ones. Acc shows some variation across runs, and its mean remains higher for EVER. The advantage of EVER is not specific to the single run reported in the main paper, and it holds on average over repeated runs on both datasets despite their small test sets.
Table R2. Mean ± standard deviation over three seeds (Acc / UAR / W-F1).
| Model | Emognition dataset | MDMER dataset |
|---|---|---|
| CBraMod | 0.322 ± 0.030 / 0.283 ± 0.036 / 0.272 ± 0.033 | 0.295 ± 0.009 / 0.209 ± 0.003 / 0.234 ± 0.014 |
| EVER | 0.341 ± 0.026 / 0.310 ± 0.013 / 0.301 ± 0.008 | 0.321 ± 0.029 / 0.225 ± 0.014 / 0.269 ± 0.021 |
Leave-one-subject-out evaluation would require one training run per held-out subject for every model on both datasets, which is not feasible within the rebuttal period. The paper also reports bootstrap confidence intervals in Tab. A3 of the appendix. Extending this analysis, a paired bootstrap on identical MDMER test trials shows that EVER exceeds CBraMod with statistically significant differences in all three metrics, as shown in Tab. R3.
Table R3. Paired comparison on the MDMER dataset over the 475 unique test trials (two-sided paired-bootstrap p-value of the difference from EVER in parentheses).
| Model | Acc (p-value) | UAR (p-value) | W-F1 (p-value) |
|---|---|---|---|
| EVER (VideoMAE + CBraMod) | 0.354 | 0.237 | 0.280 |
| CBraMod (best EEG) | 0.302 (0.012) | 0.208 (0.007) | 0.219 (0.001) |
| Logit-averaging ensemble of CBraMod and VideoMAE | 0.306 (0.019) | 0.208 (0.011) | 0.217 (0.001) |
Q3. Majority-class and chance levels
With five classes per target, uniform chance is 0.20 in accuracy. The majority-class predictor always outputs the most frequent class of the test set for each target, the strongest input-independent prediction. Its accuracy equals the share of that class. Its UAR is 0.20 because one class has recall 1 and the other four have recall 0. Its W-F1 is because only the predicted class has a nonzero F1 score. As for every model in the paper, each metric is computed per target and then averaged over valence and arousal. The results are shown in Tab. R4. EVER with VideoMAE + CBraMod exceeds this predictor on each target of both datasets, by 7.1 points on Emognition and by 3.7 points on MDMER in average accuracy, and also in UAR and W-F1.
Table R4. Majority-class predictor and EVER (VideoMAE + CBraMod).
| Emognition dataset | MDMER dataset | |
|---|---|---|
| Majority-class Acc (valence / arousal) | 0.282 / 0.318 | 0.335 / 0.299 |
| Majority-class average (Acc / UAR / W-F1) | 0.300 / 0.200 / 0.139 | 0.317 / 0.200 / 0.153 |
| EVER Acc (valence / arousal) | 0.388 / 0.353 | 0.377 / 0.331 |
| EVER average (Acc / UAR / W-F1) | 0.371 / 0.353 / 0.353 | 0.354 / 0.237 / 0.280 |
Q4. Logit-averaging ensemble
We added the logit-averaging ensemble of the same backbones as EVER for both backbone combinations on both datasets. The results are shown in Tab. R5. EVER outperforms the ensemble in all three metrics for both combinations on both datasets, and on MDMER with VideoMAE + CBraMod the gains are statistically significant in all three under the paired bootstrap, as shown in Tab. R3. This shows that the graph-based fusion contributes beyond combining two strong backbones.
Table R5. Logit-averaging ensemble versus EVER with the same backbones (Acc / UAR / W-F1, over the unique test trials of each dataset).
| Backbones | Model | Emognition dataset | MDMER dataset |
|---|---|---|---|
| CBraMod + AdaMAE | Ensemble | 0.353 / 0.288 / 0.294 | 0.300 / 0.203 / 0.212 |
| CBraMod + AdaMAE | EVER | 0.371 / 0.306 / 0.298 | 0.341 / 0.231 / 0.274 |
| CBraMod + VideoMAE | Ensemble | 0.359 / 0.292 / 0.298 | 0.306 / 0.208 / 0.217 |
| CBraMod + VideoMAE | EVER | 0.371 / 0.353 / 0.353 | 0.354 / 0.237 / 0.280 |
Response to Reviewer qnJ1 (3/3) - Fusion baselines and complementarity (Q5)
Q5. Why fusion baselines and single-component removals fall below EEG-only
On Emognition, EEG is far stronger than video (0.353 for CBraMod versus at most 0.307 for the video models in Tab. 1 of the main paper), and fusing the weaker modality without structure pulls the prediction toward it. All fusion variants fine-tune the same large self-supervised backbones with the same loss and hyperparameters as EVER, and the same recipe applied to each backbone alone yields the strongest unimodal models of the benchmark, CBraMod among the EEG models and VideoMAE and AdaMAE among the video models. The degradation therefore comes from the fusion operators rather than from the optimization. The Emognition comparison is reported in Tab. 5 of the main paper. On MDMER, concatenation, cross-attention and FiLM collapse to the majority-class predictor for both video backbones, whereas EVER does not, as shown in Tab. R6.
Table R6. Fusion strategies with identical backbones on the MDMER dataset.
| Fusion (Acc / UAR / W-F1) | AdaMAE + CBraMod | VideoMAE + CBraMod |
|---|---|---|
| Majority-class predictor | 0.317 / 0.200 / 0.153 | 0.317 / 0.200 / 0.153 |
| Concatenation | 0.317 / 0.200 / 0.153 | 0.317 / 0.200 / 0.153 |
| Cross-attention | 0.317 / 0.200 / 0.153 | 0.317 / 0.200 / 0.153 |
| FiLM | 0.317 / 0.200 / 0.153 | 0.317 / 0.200 / 0.153 |
| Pooled-node GCN | 0.320 / 0.202 / 0.167 | 0.296 / 0.195 / 0.156 |
| EVER (ours) | 0.341 / 0.231 / 0.274 | 0.354 / 0.237 / 0.280 |
This also shows why each component of EVER matters. Vector-level fusion mixes a strong and a weak modality in a single step, so the weaker one dilutes the stronger one. EVER instead turns each modality into structured nodes, with BAR-GCN producing anatomical region nodes and VCG producing temporally ordered clip nodes. The inter-modal graph lets every clip exchange information with the global EEG embedding and the region nodes under the edge mask, so that the video evidence refines the EEG representation instead of diluting it. Removing the positional encoding or the Gaussian prior takes away the temporal structure of the clip nodes, which explains the drops in Tab. 4 of the main paper.
Complementarity analysis (raised in the weaknesses)
The reviewer notes that EVER keeps only 13 of the 111 arousal samples that EEG alone classifies correctly. The predictions in Fig. A1 were taken from a different evaluation run of the unimodal baselines than the one in Tab. 1 of the main paper, and the figure will be regenerated. On MDMER arousal, the video-only model concentrates almost all of its predictions on the majority class, whereas EVER predicts four distinct classes, so the fused model does not simply follow the video prediction.
Response to Reviewer qnJ1 (1/3) - Task context and early stopping (Q1)
We first apologize that the number of evaluation samples in the submission was incorrect because of an implementation issue in the distributed evaluation. Please kindly refer to our response to Reviewer k3Z3 (W1(a) and Q1) for the details. Accordingly, the results in this response are updated to the unique test trials and may differ slightly.
We thank the reviewer for acknowledging the fusion design, the montage-flexible region mapping and the statistical candor of the paper. In brief, we add (i) an evaluation with held-out validation subjects for Q1, (ii) repeated runs over three seeds for Q2, (iii) the majority-class and chance levels for Q3, (iv) a logit-averaging ensemble of the same backbones for Q4, and (v) a fusion comparison with identical backbones on MDMER for Q5. The questions cover the points raised in the weaknesses, so we respond in the order of the questions and address the remaining point on the complementarity analysis at the end.
Context on the task
Subject-independent emotion recognition from EEG and facial video differs from a standard vision task in three ways. First, the labels are self-reported valence and arousal ratings of each trial, which are subjective and vary in scale across participants. Second, EEG shows strong inter-subject variability, so a model must generalize to unseen people rather than to unseen images. Third, participants watch the stimuli passively and often show only subtle facial expressions, and the paired EEG–video datasets are small, with about 400 trials in Emognition and 2.3K trials in MDMER in total. For these reasons, accuracies on five-class valence and arousal remain modest even for large pretrained models. In our benchmark, the EEG foundation models CBraMod, LaBraM, REVE and BIOT all stay between 0.25 and 0.36 average accuracy. A method should thus be assessed by its margin over the trivial and the strongest unimodal baselines on identical samples.
Q1. Set used for early stopping
Every model in the benchmark is reported at its best epoch on the evaluation split, under one identical protocol for all 10 unimodal backbones on both datasets and for all fusion models. The video encoders VideoMAE and AdaMAE are pretrained by self-supervised masked autoencoding on large unlabeled video collections, and only their final encoder block and the classification head are fine-tuned (Section 3.1.1). The EEG foundation models are pretrained on large EEG corpora and fine-tuned with their downstream setups. In the subject-independent setting, the models start to overfit the training subjects within a few epochs, and the evaluation loss rises thereafter, so we reported the best epoch identically for every model.
To examine whether this selection creates the advantage of EVER, we separate the model selection from the test subjects. Eight validation subjects (256 trials) are held out from the training split and are used for model selection. The test subjects are identical to the main paper and are used only for the final evaluation. The evaluation is conducted on MDMER, where the larger number of subjects permits a validation set of this size. It covers EVER and its two unimodal backbones, CBraMod and VideoMAE. The results over three seeds are shown in Tab. R1.
Table R1. Evaluation with held-out validation subjects on the MDMER dataset (mean ± standard deviation over three seeds).
This protocol is stricter than the main paper, because the model selection is performed without access to the test subjects. Compared with the three-seed results selected on the evaluation split in Tab. R2, the accuracy decreases by a similar amount for both models, by 0.055 for EVER and by 0.050 for CBraMod. This indicates that the selection does not create the gap between them. EVER remains higher than both of its unimodal backbones in all three metrics. The gain is clearest on UAR and W-F1, the two metrics that account for class imbalance. For instance, W-F1 reaches 0.253 against 0.211 for CBraMod and 0.247 for VideoMAE, and UAR reaches 0.221 against 0.201 and 0.212. We will include this result in the main manuscript.