- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes HB-IGCL, a self-supervised pretraining framework for emotion recognition from long ECG–EEG recordings. Instead of assigning the recording-level label to every short segment, each recording is split into non-overlapping instances and treated as a bag. Two contrastive objectives align local instances of one modality with the global representation of the other modality from the same recording, using other recordings in the batch as negatives. C2E contrasts ECG instances with the averaged EEG global representation. E2C contrasts EEG instances with an ECG global representation weighted by the C2E scores. A spatial-guided alignment module (SGAM) additionally aligns a lead-level spatial representation with the global temporal one. The framework uses four encoders (two per modality), a proposed multi-branch instance encoder (AMIE), and a gate-and-refinement (GR) module. For downstream classification, the pretrained encoders are frozen and combined through a cross-attention head and a classifier. Experiments on AMIGOS, DREAMER (4-class), and MPED (3-class) under subject-independent splits compare HB-IGCL with three pretraining frameworks (LMM, EMM, CCM) across five encoders. Among these frameworks, HB-IGCL gives the best F1 for every encoder.
Critical strengths
-
The motivation is reasonable. Assigning the recording-level label to every short segment is a real issue for long physiological recordings, and using the within-recording ECG–EEG correspondence as a training signal is a sensible alternative.
-
The experimental effort is substantial: five encoders, three datasets, five seeds, and subject-independent evaluation. The authors also adapt the baselines to their setting and report the full sweeps (App. F–H) rather than only the selected configurations, which makes the comparison more transparent.
Critical weaknesses
-
The gain from pretraining is inconsistent across datasets. Compared with training the same model from scratch (Rand. Init., Table 2), frozen HB-IGCL features improve F1 clearly on AMIGOS (+9.9) and modestly on DREAMER (+3.8), but are slightly worse on MPED (84.66 vs. 85.86). With full fine-tuning, the differences are +5.6, −0.5, and +3.7, mostly within the reported standard deviations. Pretraining does reduce variance in several settings, but part of the performance seems to come from the encoder and downstream architecture. A supervised-from-scratch row for each encoder in Table 1 would make the contribution of pretraining clearer.
-
Role of the fusion head. In Table 3, the single-modality variants are close to chance, and concatenating both modalities gives similar results (27.45 F1 on AMIGOS, 42.24 on MPED), while the cross-attention head reaches 66.23 and 84.66. Y_c and Y_e are described as representation vectors (L269–273). If cross-attention is computed over a single key token, it reduces to a projection of the EEG features, so it is unclear what produces such a large gap. Clarifying the head design, its parameter count, and whether the single-modality variants use the same frozen encoders would help interpret how much of the result reflects the pretrained representations.
-
Inconsistent results across tables. The “GR” rows of Table 4 (DREAMER) report 42.47 / 60.55 (AMIE), 24.80 / 46.39 (ModernTCN) and 15.92 / 32.22 (PatchTST). Table 1 reports 50.53 / 63.61, 31.73 / 50.00, and 28.99 / 44.44 for what appears to be the same configuration. I may be missing a difference in setup, but none is described. Within Table 4, “w/o GR” also matches or beats “GR” in F1 for ModernTCN and OS-CNN, so the evidence for GR seems mixed.
-
Model selection. App. C and H state that α = 0.5, β = 0.8, and the CCM coefficient a = 0.3 were chosen because they give the best results in the reported sweeps (L732–741, L884–885). These values are identical, including standard deviations, to the corresponding test results in Table 1 (e.g., Table 10 vs. CCM-AMIE: 56.07±2.99 / 57.69±2.81). T = 4 s is likewise chosen from Fig. 9. Since a validation split exists, please clarify whether these selections were made on the validation or the test subjects.
-
Limited evaluation protocol. Each dataset uses a single fixed subject split with few test subjects (5 / 4 / 3; Table 5), and the five seeds only vary the initialization, so the standard deviations do not reflect variation across subjects. K-fold cross-subject or leave-one-subject-out evaluation would make the results more reliable.
-
Minor
- The instance-removal analysis (Fig. 6) is on AMIGOS only, without error bars, and is non-monotonic.
Questions for authors
-
Were α, β, T, and the baseline hyperparameters selected on the validation or the test subjects?
-
What setup produces the Table 4 “GR” results, and why do they differ from Table 1?
-
How does a supervised-from-scratch model perform with each encoder in Table 1?
-
How is the cross-attention head defined? Are Y_c and Y_e single tokens, and how many parameters does it have compared with the concatenation head?
Response to Reviewer CQ9D
We thank the reviewer for the thoughtful feedback. New results will be added to the revised manuscript.
1. W1 + Q3 (Supervised-from-scratch controls)
We have added supervised-from-scratch controls for every encoder (Table 8, Appendix E of the revised manuscript); the table reports supervised-from-scratch test F1 and Acc, to be compared with Table 1.
2. W2 + Q4 (Definition of the fusion head)
Neither Yi,c nor Yi,e is a single token. An ECG signal is first split into instances, each fed into two parallel branches (ECG Encoder + GR module + GAP) that each output a 1×128 tensor; concatenating the two branch outputs yields the final ECG comprehensive representation Yi,c ∈R2×128. The EEG signal is processed analogously to obtain Yi,e ∈R2×128. We have clarified this in the revised manuscript.
The “concatenation + classifier” head has 2,052 trainable parameters, and the cross-attention head has 199,300 only 2.62% of the model’s ~7.6M trainable parameters), while the performance gain is substantial: F1 rises from 27.45% to 66.23% on AMIGOS and from 42.24% to 84.66% on MPED. All unimodal and multimodal variants in Table 3 use the same pretrained and frozen encoders and the same data split, differing only in fusion strategy; the Table 3 caption has been revised accordingly.
3. W3 + Q2 (The setting behind the GR results in Table 4)
The discrepancy between Tables 4 and 1 stems from the protocol. The original Table 4 reports results with ECG and EEG concatenated before the encoder, without pretraining and with the concatenated input passed directly to the encoder, serving as a quick check of the GR design; Table 1 reports pretraining followed by fine-tuning. The table below reports the GR ablation with pretraining and fine-tuning and replaces the original Table 4 in the revised manuscript.
4. W4 + Q1 (Model selection: validation or test set)
The hyperparameters (α,β,T) were selected on the validation set, and they also yield the best overall performance on the test set (Figures 8 and 9, original manuscript). The corresponding validation-set sweeps, which are consistent with the test-set optima, have been added to Appendices C and D of the revised manuscript as Tables 6 and 7.
The hyperparameters of the baseline models correspond to the best results on the test set; the full results are reported in Tables 7–10 of the original appendix (Tables 10–13 in the revised manuscript). Baseline hyperparameters are set to yield their best test-set results, making the evaluation of HB-IGCL more stringent.
5. W5 (Single fixed split and few test subjects)
Since the three datasets contain only 40, 23, and 23 subjects and all suffer from class imbalance (severe in AMIGOS and DREAMER), random subject-independent splitting may leave certain classes absent from the validation or test subsets. We therefore follow [1] and [2] in assigning subjects to mutually exclusive training, validation, and test subsets. We have added these details to Appendix A of the revised manuscript.
Reference:
[1] Xu et al., IEEE BIBM 2025.
[2] Zhong et al., IEEE BIBM 2025.