- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Addressing the need for EEG analysis in Infantile Epileptic Spasms Syndrome (IESS), this paper introduces the IESSEEG open dataset. The dataset comprises 150 raw long-term EEG recordings from 100 pediatric subjects—totaling approximately 3,172 hours—and includes diagnostic labels, treatment response labels, clinician annotations, BASED scores, and relevant clinical metadata. Based on this resource, the authors established benchmarks for IESS diagnosis and treatment response prediction, systematically comparing traditional quantitative EEG (qEEG) methods with eight EEG foundation models; they further analyzed the complementary roles of full-recording qEEG features and clinical metadata in predicting treatment response. The study offers significant clinical research value and contributes a valuable data resource. However, there remain certain issues that require further refinement.
Critical strengths
- The research motivation is clear.
- The experimental design is rigorous.
- It offers clinical research value and contributes data resources.
Critical weaknesses
- The abstract mentions that combining qEEG, foundation model features, and clinical metadata can improve the prediction of treatment response. It is recommended to include a key quantitative metric here to enhance the abstract's objective persuasiveness and scientific appeal.
- In Table 1, the median age at EEG acquisition was 0.93 years for the case group and 1.52 years for the control group, with age ranges of 0.37–8.31 years and 0.10–17.73 years, respectively; this indicates a noticeable discrepancy. Given that background rhythms, spectral distribution, and neurodevelopmental characteristics in pediatric EEGs change significantly with age, this age mismatch might lead the diagnostic model to learn age-related features rather than patterns specifically indicative of IESS abnormalities. The authors are advised to conduct age-matching or age-stratified analyses and compare diagnostic performance before and after matching to assess potential age-related confounding effects in the dataset.
- Section 3.3 describes how seven clinicians independently scored 20 raw long-term EEG recordings, resulting in a total of 140 BASED scoring records. However, the proportion of records with BASED scores is relatively low compared to the 150 raw long-term EEG recordings in the dataset. The authors should clarify the criteria used to select these 20 recordings, the distribution of BASED score grades, and the changes in scores before and after treatment. Additionally, reporting inter-rater reliability would better demonstrate the quality of this annotated resource and clarify whether it is best suited for model validation, studies on scoring consistency, or training automated BASED scoring models.
- The study employed a fixed 5-fold cross-validation approach. Given the limited dataset size—comprising only 100 patients, with the response group containing just 50 individuals (28 vs. 22)—it is recommended to perform multiple splits using different random seeds or to employ bootstrapping for confidence interval estimation to ensure more robust statistical results.
- The conclusion summarizes the study's contributions regarding the construction of the IESSEEG open dataset and the prediction of treatment response; however, the discussion of study limitations lacks sufficient focus. It is recommended that the author add a dedicated discussion on limitations at the end of Chapter 6, or supplement the current limitations of IESSEEG in Chapter 7, to provide direction for future data collection aimed at generalization within the field.
Questions for authors
None
6 comments
Response to Reviewer KC9x (2/4)
Critical Weaknesses 2 – Diagnosis in the age-matched subgroup (continued)
Table 1. IESS diagnosis task results against clinician diagnosis labels. AUROC, balanced accuracy and 95% confidence intervals are calculated as in Section 4.4 and Appendix B.6. The full cohort has 100 patients (50 cases and 50 controls) and 200 routine clips. The age-matched subgroup has 60 patients (40 cases and 20 controls) and 120 routine clips.
Clinician diagnosis labels
| Method | Full cohort AUROC | Full cohort Balanced accuracy | Age-matched AUROC | Age-matched Balanced accuracy |
|---|---|---|---|---|
| qEEG triplet | 90.7 [85.9–95.4] | 83.5 [77.9–89.2] | 93.9 [88.7–99.2] | 78.1 [70.4–85.8] |
| Expanded qEEG | 94.7 [91.3–98.1] | 86.2 [80.7–91.7] | 97.0 [93.7–100.0] | 86.7 [80.3–93.1] |
| Frozen encoders | ||||
| BIOT | 88.4 [82.9–94.0] | 81.5 [75.7–87.3] | 88.4 [81.3–95.6] | 77.6 [69.2–85.9] |
| LaBraM | 97.3 [95.3–99.4] | 89.2 [83.9–94.4] | 95.5 [91.7–99.3] | 85.2 [78.7–91.6] |
| CBraMod | 92.2 [87.8–96.6] | 85.1 [79.6–90.6] | 92.6 [86.9–98.3] | 81.7 [74.3–89.2] |
| EEGPT | 93.2 [88.9–97.5] | 84.7 [78.5–90.8] | 91.5 [84.5–98.5] | 81.9 [72.7–91.1] |
| LUNA | 88.1 [81.6–94.6] | 84.7 [78.8–90.5] | 89.2 [82.4–96.0] | 84.5 [76.8–92.1] |
| REVE | 91.1 [86.3–96.0] | 85.5 [79.9–91.1] | 94.2 [88.8–99.7] | 85.8 [77.8–93.8] |
| CodeBrain | 95.1 [91.9–98.4] | 89.9 [84.7–95.1] | 93.2 [87.8–98.6] | 83.1 [75.3–90.9] |
| CSBrain | 90.2 [84.8–95.6] | 81.1 [75.2–86.9] | 89.2 [82.0–96.4] | 81.7 [74.1–89.4] |
| Fully fine-tuned encoders | ||||
| BIOT | 95.0 [91.7–98.2] | 86.6 [81.2–92.0] | 91.7 [86.2–97.2] | 81.8 [72.5–91.1] |
| LaBraM | 97.4 [94.9–99.9] | 92.5 [88.2–96.7] | 92.9 [86.6–99.2] | 85.3 [76.3–94.2] |
| CBraMod | 95.2 [91.4–99.0] | 90.0 [85.4–94.6] | 94.7 [89.4–100.0] | 89.1 [82.3–96.0] |
| EEGPT | 92.6 [88.1–97.1] | 87.0 [81.4–92.6] | 91.1 [83.6–98.6] | 80.3 [70.9–89.8] |
| LUNA | 92.4 [87.2–97.7] | 85.8 [80.0–91.6] | 89.9 [83.1–96.7] | 78.3 [68.3–88.3] |
| REVE | 91.4 [86.6–96.3] | 84.7 [78.6–90.8] | 93.4 [88.1–98.8] | 83.1 [74.3–92.0] |
| CodeBrain | 96.2 [93.0–99.5] | 90.5 [85.9–95.2] | 96.0 [92.1–99.9] | 83.8 [75.0–92.7] |
| CSBrain | 96.4 [93.2–99.6] | 92.7 [88.8–96.7] | 94.9 [89.5–100.0] | 88.7 [81.2–96.3] |
Critical Weaknesses 3 – BASED coverage, score distribution and reliability
The BASED annotations in IESSEEG come from the reliability study by Swarz et al. [1] (Section 3.3). In the study, ten pre-treatment and ten post-treatment long EEG recordings from 20 different patients were randomly selected for annotation. Four experts and three trainees independently reviewed all 20 recordings and assigned a BASED score to each, yielding 140 ratings. Scoring followed the 2021 BASED criteria [2, 3].
Please refer to Table 2 for the distribution of released BASED scores. Pre-treatment ratings are concentrated at grade 5, whereas post-treatment ratings are more dispersed. Because these 20 recordings come from different patients, the table compares pre-treatment and post-treatment score distributions across groups. Within-patient changes after treatment cannot be assessed.
Table 2. BASED grade counts across seven ratings per recording.
| Recording group | BASED score 0 | BASED score 1 | BASED score 2 | BASED score 3 | BASED score 4 | BASED score 5 | Total |
|---|---|---|---|---|---|---|---|
| Pre-treatment | 0 | 0 | 3 | 4 | 8 | 55 | 70 |
| Post-treatment | 28 | 8 | 11 | 10 | 6 | 7 | 70 |
Following Mytinger et al. [3], we computed the intraclass correlation coefficient (ICC) from all 140 original ratings. The ICC was 0.78 (95% confidence interval 0.65–0.89) across all seven raters, 0.85 (0.73–0.93) for experts and 0.68 (0.45–0.84) for trainees. Both the overall ICC of 0.78 and the expert-rater ICC of 0.85 fall within the good-agreement range. These values are comparable to the ICC of 0.86 reported in the recent multicenter BASED validation study by Mytinger et al. [3], supporting the reliability of our BASED annotations. We provide these annotations as a supplementary resource to support exploratory research and analysis by IESSEEG users and the broader research community.
Response to Reviewer KC9x (3/4)
Critical Weaknesses 4 – Confidence intervals and bootstrap analysis
We added patient-level bootstrap results and confidence intervals for diagnosis and response prediction. Keeping the experimental settings unchanged, we resampled patients within each test fold 1,000 times. Tables 3 and 4 report bootstrap mean AUROC and balanced accuracy with 95% percentile confidence intervals.
Table 3. Diagnosis AUROC and balanced accuracy (%). Entries are bootstrap means with 95% percentile confidence intervals.
Patient diagnosis labels
| Method | Full cohort AUROC | Full cohort Balanced accuracy | Age-matched AUROC | Age-matched Balanced accuracy |
|---|---|---|---|---|
| qEEG triplet | 91.4 [87.1–95.7] | 85.5 [80.0–91.0] | 94.3 [89.7–97.8] | 80.3 [72.4–87.6] |
| Expanded qEEG | 93.2 [87.9–97.2] | 85.6 [79.7–90.8] | 93.3 [87.5–98.2] | 83.2 [76.2–89.5] |
| Frozen encoders | ||||
| BIOT | 87.6 [80.5–93.6] | 81.5 [75.6–87.0] | 85.5 [74.3–94.8] | 76.8 [66.7–86.1] |
| LaBraM | 94.8 [91.1–98.1] | 86.6 [80.7–92.0] | 94.2 [87.9–98.8] | 81.2 [73.5–87.8] |
| CBraMod | 90.7 [84.7–96.0] | 84.5 [79.2–89.6] | 92.5 [85.1–98.0] | 81.9 [74.4–88.5] |
| EEGPT | 94.6 [89.7–98.0] | 87.4 [81.5–92.3] | 94.5 [88.3–99.0] | 84.1 [75.9–91.8] |
| LUNA | 87.5 [79.9–93.8] | 84.2 [78.2–89.4] | 88.2 [80.3–95.4] | 82.3 [75.1–89.4] |
| REVE | 92.7 [87.1–97.0] | 87.5 [81.9–92.4] | 95.5 [89.5–99.6] | 88.7 [82.7–94.3] |
| CodeBrain | 94.6 [90.4–97.8] | 88.2 [81.9–93.4] | 91.6 [84.4–97.6] | 81.2 [75.2–87.5] |
| CSBrain | 88.0 [81.3–93.4] | 80.6 [74.3–85.9] | 85.2 [76.1–93.2] | 78.3 [70.1–86.1] |
| Fully fine-tuned encoders | ||||
| BIOT | 97.1 [94.2–99.1] | 90.5 [85.4–94.5] | 91.8 [84.0–98.0] | 87.0 [77.5–95.1] |
| LaBraM | 97.5 [94.6–99.6] | 93.4 [89.0–97.2] | 96.5 [91.3–99.7] | 88.6 [79.9–96.3] |
| CBraMod | 96.2 [92.1–99.4] | 90.9 [85.6–95.4] | 97.8 [94.7–100.0] | 91.2 [85.4–96.2] |
| EEGPT | 95.3 [91.0–98.5] | 91.0 [86.4–95.3] | 94.1 [87.3–99.0] | 79.1 [69.2–88.4] |
| LUNA | 93.8 [88.0–98.5] | 87.0 [81.1–92.2] | 91.6 [84.8–97.3] | 79.7 [68.7–89.5] |
| REVE | 94.6 [90.1–98.0] | 88.3 [82.8–93.2] | 94.6 [88.7–99.0] | 87.8 [79.3–95.4] |
| CodeBrain | 96.0 [92.5–98.7] | 90.4 [85.1–94.6] | 98.1 [95.4–100.0] | 84.7 [74.4–93.3] |
| CSBrain | 96.5 [93.1–99.1] | 91.1 [86.3–95.2] | 95.8 [90.4–99.9] | 90.4 [83.1–96.8] |
Table 3. Diagnosis AUROC and balanced accuracy (%). Entries are bootstrap means with 95% percentile confidence intervals.
Clinician diagnosis labels
| Method | Full cohort AUROC | Full cohort Balanced accuracy | Age-matched AUROC | Age-matched Balanced accuracy |
|---|---|---|---|---|
| qEEG triplet | 90.6 [85.5–95.2] | 83.6 [78.1–89.2] | 94.0 [87.8–98.6] | 78.4 [70.7–85.9] |
| Expanded qEEG | 94.7 [90.6–97.8] | 86.2 [81.0–91.2] | 97.0 [92.6–99.8] | 86.9 [80.4–92.5] |
| Frozen encoders | ||||
| BIOT | 88.4 [82.0–94.0] | 81.5 [75.6–87.1] | 88.2 [79.8–95.2] | 77.6 [68.7–85.6] |
| LaBraM | 97.4 [95.0–99.2] | 89.2 [83.9–93.9] | 95.6 [91.2–98.7] | 85.2 [78.2–91.1] |
| CBraMod | 92.3 [86.9–96.4] | 85.1 [79.7–89.9] | 92.4 [85.3–97.6] | 81.9 [74.3–88.5] |
| EEGPT | 93.0 [87.9–97.0] | 84.6 [78.5–90.1] | 91.6 [83.9–98.0] | 81.8 [72.9–90.0] |
| LUNA | 88.1 [80.4–94.5] | 84.7 [78.7–89.8] | 89.2 [81.7–96.2] | 84.5 [76.8–91.3] |
| REVE | 91.1 [85.8–95.6] | 85.4 [80.0–90.5] | 94.3 [88.2–99.3] | 86.0 [77.6–92.7] |
| CodeBrain | 95.1 [91.4–98.2] | 90.0 [84.5–95.0] | 93.4 [87.0–98.2] | 83.3 [75.2–90.3] |
| CSBrain | 90.2 [83.6–95.3] | 81.1 [75.3–86.6] | 89.2 [80.8–95.9] | 82.0 [74.0–88.9] |
| Fully fine-tuned encoders | ||||
| BIOT | 94.9 [91.0–97.8] | 86.6 [81.1–91.7] | 91.6 [84.3–96.9] | 81.9 [72.5–90.8] |
| LaBraM | 97.3 [94.3–99.5] | 92.4 [88.3–96.0] | 93.0 [86.0–98.6] | 85.2 [76.4–93.2] |
| CBraMod | 95.0 [90.4–98.4] | 89.8 [85.1–94.0] | 94.8 [88.3–99.2] | 89.2 [82.4–94.9] |
| EEGPT | 92.4 [87.4–96.5] | 87.0 [81.9–92.0] | 91.1 [83.1–97.9] | 80.2 [71.8–89.0] |
| LUNA | 92.3 [86.1–97.0] | 85.8 [79.6–91.0] | 89.9 [82.0–96.1] | 78.3 [68.1–87.2] |
| REVE | 91.2 [85.7–95.5] | 84.5 [78.8–90.4] | 93.4 [87.1–98.4] | 83.1 [73.9–91.3] |
| CodeBrain | 96.1 [92.1–98.9] | 90.5 [85.2–94.9] | 95.9 [90.6–99.4] | 83.7 [74.7–91.3] |
| CSBrain | 96.2 [92.3–98.8] | 92.8 [88.7–96.4] | 94.8 [87.8–99.2] | 88.8 [80.6–95.5] |
Response to Reviewer KC9x (1/4)
We thank the reviewer for the detailed assessment and constructive suggestions. In response, we repeated the diagnostic benchmark in the age-matched subgroup and added patient-level bootstrap analysis. We also report the BASED score distribution and inter-rater reliability, and provide content we will add to the abstract and limitations discussion.
Critical Weaknesses 1 – Quantitative result in the abstract
We will add the following content to the abstract:
Critical Weaknesses 2 – Diagnosis in the age-matched subgroup
Diagnostic discrimination remained high in the age-matched subgroup. Fully fine-tuned foundation models achieved mean five-fold AUROCs of 91.7–98.1% against patient diagnosis labels and 89.9–96.0% against clinician diagnosis labels. We evaluated the age-matched subgroup of 40 cases and 20 controls in the metadata, using the same benchmark settings as in Section 4 and Appendix B.
For the age-matched subgroup, median acquisition age is 0.91 years (interquartile range 0.64–1.81) for cases and 0.83 years (0.62–1.95) for controls. Mean ages are 1.72 years in both groups. Table 1 compares the full-cohort and subgroup results against both patient and clinician diagnosis labels.
Table 1. IESS diagnosis task results against patient diagnosis labels. AUROC, balanced accuracy and 95% confidence intervals are calculated as in Section 4.4 and Appendix B.6. The full cohort has 100 patients (50 cases and 50 controls) and 200 routine clips. The age-matched subgroup has 60 patients (40 cases and 20 controls) and 120 routine clips.
Patient diagnosis labels