- Rating
- 1
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper points out the challenge of cross-subject EEG decoding and provides a general framework by domain-incremental learning, treating each new subject as a domain and adapting sequentially through replay memory. By comprehensively design the memory construction and the virtual subject generation, this paper gives clear framework on how do sequentially adapting EEG models for upcoming subjects. The evaluation metrics and datasets are also comprehensively defined and presented, and give clear result and analysis.
Critical strengths
Overall, the domain-incremental learning framework proposed in this work is elegant, well-defined, and general for cross-subject EEG decoding. The issue they want to solve is real and well motivated. The framework and the algorithm they propose are well-written and well-designed. The elaboration and the visualization are also well presented. The paper is well structured and friendly to follow. Everything before Section III looks strong to me.
Critical weaknesses
Major issues start to surface in Section III. First, the performance is just not good enough to make it play its role in the field. Second, the fundamental setup on the EEG FM they used is not clearly stated, which leaves the number in danger to represent the real behavior of how EEG FM would perform. The only major impact I can find after reading the section III is about the privacy de-identification potential that can be provided by the virtual subject, but this paper does not quantify and analyze the privacy aspect in the current manuscript.
Questions for authors
- In the current section III (Experiments), I don’t see how the author pre-train the EEG FM CBraMod. By the current description in line 347, readers would wonder: is CBraMod just pretrained on the “source subjects” they defined in table 1, without pretraining on any large corpus? If so, this is not a fully pretrained foundation model. This is an empty foundation model pre-trained with a super tiny pre-trained dataset (like N_s in ). The original CBroMod is pretrained on 9000 hours of EEG data. If the CBraMod is pre-trained using the official recipe, or just downloaded from the official release, please state it clearly. I assume this paper use CBraMod pretrained on the official recipe, and ask the following questions.
- Please calibrate the performance of Oracle (joint training) with the official performance of CBraMod on their fine-tuned datasets, if they are listed in the official paper, like PhysioNet-MI, SHU-MI, and FACED. If the number does not match, please specify and explain possible cause and difference in the implementation. The performance reported in this paper as Oracle (joint training) should match the number reported in the CBraMod's fine-tuned result.
- The current improvement of WAVE, compared to other baselines, like random replay, EWC, and LwF, are just too weak and marginal. Even on FACED, the observed outperfomed margin is not salient. Even though this paper has 4 seeds for each experiment, does each seed also randomly pick the subjects in source, incremental, and generalization domain? If not, the current numbers of subjects in each domain are just too small to make the result stable. In BCI-IV 2a, only 2 in source, 5 in incremental, and 2 in generalization domain. What if the 2 subjects in the generalization just happened to be supreme under WAVE? So, the subject pool should be shuffled to be reported, especially when the performance improvement is marginal.
- Only one FM is used. Since the performance margin of WAVE on CBraMod is too small compared to other baselines (random replay, EWC, and LwF), it is likely that such margin may disappear when applied on other EEG FM. To make this paper citable, this method should also be tested on other EEG FMs. This is not to ask the author to pre-train each FM from scratch, but to just download the official release of each EEG FM, and further fine-tune on the small source and incremental domain they define.
- How does this method perform on small and task-oriented model? For the SOTA task-oriented models on each of the dataset tested in this manuscript (BCI-IV-2a, PhysioNet-MI, SHU-MI, FACED, ISRUC), can each dataset's SOTA method also be benefited from the proposed domain-incremental learning framework?
- Does the privacy of each subject can actually be de-identified via the virtual subject augmentation? If so, this adds value to this paper, as the current performance analysis does not show satisfactory impact for this paper.