- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes KoMaNet (Koopman Manifold Network) for EEG-based objective assessment of stress resilience, aiming to capture stable, long-term psychological traits from transient brain activity. The framework consists of three components: a Self-supervised State Network for learning stable brain-state representations, a Koopman Dynamic Network for modeling state evolution and deviations from stable dynamics, and a Manifold Subject Network for subject-level prediction through distributional representations. The authors also introduce RESIST, a self-collected dataset comprising 160 subjects and approximately 630 hours of EEG recordings across overnight-sleep and afternoon-nap paradigms. Experiments show that KoMaNet outperforms existing EEG and Koopman-based baselines, achieving AUCs of 0.740 and 0.700 on RESIST Night and RESIST Day, respectively. Ablation studies, visualization analyses, and additional experiments on the combined RESIST setting and external LEMON dataset further evaluate the effectiveness and generalizability of the framework.
Critical strengths
- Clear motivation and reasonable methodological design. The paper addresses an interesting problem: predicting a relatively stable psychological trait from EEG signals that primarily reflect transient brain activity. The three components of KoMaNet are designed to address different aspects of this challenge, from learning stable state representations to modeling temporal dynamics and aggregating subject-level information. The ablation results show that each component contributes to the final performance, providing reasonable support for the proposed design.
- A new dataset and encouraging experimental results. The authors collect RESIST, containing 160 subjects and approximately 630 hours of EEG recordings, which is valuable given the limited availability of EEG datasets with stress resilience annotations. KoMaNet achieves AUCs of 0.740 and 0.700 on RESIST Night and RESIST Day, outperforming the strongest baselines by 0.070 and 0.104, respectively. The additional ablation and visualization experiments also help support the effectiveness of the framework.
Critical weaknesses
- Concerns about the robustness of KoMaNet. KoMaNet achieves AUCs of 0.740 and 0.700 on RESIST Night and RESIST Day, respectively, but its performance drops to 0.577 in the combined RESIST experiment (Appendix B), only slightly higher than the best baseline (0.569). The combined setting uses six shared electrodes and 30-minute recordings, so the reduced recording duration and differences between cohorts may partly explain this degradation. Nevertheless, this result raises concerns about the robustness of KoMaNet under different recording configurations. Further analysis would help clarify the factors behind this performance drop, particularly given the paper's emphasis on its applicability to both overnight sleep and afternoon naps.
- Impact of class imbalance on evaluation reliability. The RESIST dataset exhibits substantial class imbalance, with only 19 low-resilience subjects out of 100 in RESIST Night and 11 out of 60 in RESIST Day. Under five-fold cross-validation, each test fold therefore contains very few low-resilience subjects, making the evaluation potentially sensitive to individual predictions. Although the authors report balanced accuracy and AUC over multiple random seeds, it remains unclear how consistently KoMaNet performs on the minority class. Additional analysis of class-specific performance and variability across folds would help establish whether the reported improvements are reliable, particularly for identifying low-resilience subjects.
Questions for authors
- In Appendix B, KoMaNet's AUC drops to 0.577 in the combined RESIST setting, compared with 0.740 and 0.700 on RESIST Night and RESIST Day separately. What are the main factors behind this performance drop? Could it be related to the shorter recording duration, reduced electrode coverage, differences in the pretrained encoders, or differences between the two cohorts?
- Given the substantial class imbalance in RESIST Night (19 low-resilience vs. 81 high-resilience subjects) and RESIST Day (11 vs. 49), could the authors provide class-specific performance, such as sensitivity and specificity, along with the variability across cross-validation folds? This would help assess how reliably KoMaNet identifies low-resilience subjects.
- How was the CD-RISC-10 threshold of 20 selected for binary classification? Was it chosen as the midpoint of the scale, based on established criteria, or according to the score distributions? Have the authors examined whether the classification results are sensitive to this threshold?
- Appendix A states that the control states are determined using sleep stages, phase bins, or k-means clustering. Could the authors clarify how the sleep-stage information is obtained for RESIST Night and RESIST Day, and whether external sleep-stage annotations are required during inference?
- Figure 5 demonstrates cross-session stability using six EEG recordings from a single subject over approximately 2.5 months. Could the authors clarify whether these predictions follow the same subject-wise held-out evaluation protocol described in Section 4.2? Are repeated-session recordings available for other subjects?
- Since RESIST is presented as one of the main contributions, do the authors plan to release the dataset, including EEG recordings and resilience annotations? If public release is restricted by privacy or ethical considerations, would controlled access be possible?
5 comments
R2. Class-specific performance and fold variability (Weakness 2, Question 2).
Thanks for your constructive comments. Following your suggestion, we report sensitivity and specificity with low resilience as the positive class. Sensitivity and specificity were 0.412 ± 0.091 and 0.858 ± 0.109 on RESIST Day, and 0.557 ± 0.252 and 0.891 ± 0.033 on RESIST Night, respectively.
At these thresholds, the model missed roughly half of the low-resilience participants on average across folds and identified the larger high-resilience group more accurately in both cohorts. The small number of low-resilience participants limits how precisely we can estimate sensitivity. In future work, we plan to collect EEG recordings from more participants with low resilience.
R3. Choice of the CD-RISC-10 threshold (Question 3).
Thanks for your constructive comments. We chose 20 because it is the midpoint of the CD-RISC-10 score range (0–40). Higher scores indicate greater self-reported resilience. For our binary prediction task, we assigned scores of 20 or below to the lower-resilience group and scores above 20 to the higher-resilience group. At this cutoff, KoMaNet achieved an AUC of 0.740 on RESIST Night and 0.700 on RESIST Day (Table 1).
We also predicted continuous CD-RISC-10 scores without applying a cutoff. We evaluated these predictions using Gaussian negative log-likelihood (NLL) and empirical coverage of the central 50% prediction interval (Cov@50). NLL was 3.25 on Night and 3.36 on Day. The corresponding intervals covered 58% and 55% of observed scores, respectively (Table 1).
Following your suggestion, we evaluated two additional CD-RISC-10 cutoffs. In each fold, we calculated the mean and median questionnaire scores using only the training participants.
| CD-RISC-10 cutoff | Night BAcc | Night AUC | Day BAcc | Day AUC |
|---|---|---|---|---|
| Scale midpoint (20) | 0.724 | 0.740 | 0.635 | 0.700 |
| Training-fold mean | 0.636 | 0.710 | 0.559 | 0.638 |
| Training-fold median | 0.628 | 0.705 | 0.545 | 0.621 |
The 20-point cutoff yielded the highest BAcc and AUC on both datasets. Using the training-fold mean or median can place more participants with similar questionnaire scores on opposite sides of the cutoff, which may make the groups harder to distinguish. We use 20 as a scale-based task definition.
R1. Performance on the combined RESIST cohort (Weakness 1, Question 1).
Thanks for your constructive comments. On RESIST Night, KoMaNet predicts resilience from six hours of overnight EEG using only six electrodes (AUC 0.740). On RESIST Day, it uses 128 electrodes but requires only 30 minutes of afternoon-nap EEG (AUC 0.700).
Appendix B asks whether a more portable setup can combine the low electrode count of the overnight setting with the short recording time of the daytime setting. We evaluated a single model across both cohorts using 30 minutes of EEG from the six shared electrodes. The model achieved an AUC of 0.577, indicating that prediction remained feasible under these more restrictive conditions, although discrimination was modest and performance declined relative to both main-text settings.