- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Some EEG foundation models first convert brain signals into discrete codes with a tokenizer, then train a larger model on those codes. This paper studies how to update such a model as new EEG datasets arrive over time, without retraining from scratch. The challenge is that updating the tokenizer can change the code assigned to the same signal, which shifts the “vocabulary” the downstream model depends on. The proposed method, CAP-CL, keeps a small memory of past data and encourages only the coarsest level of codes to stay consistent with the previous version of the tokenizer, while the finer levels remain free to adapt. On a sequence of 21 public EEG datasets, CAP-CL reaches about 0.42 balanced accuracy, versus roughly 0.40–0.41 for replay and the other continual-learning baselines. The ranking holds across seeds and dataset orderings. The authors also find that forcing the codes to match the original tokenizer throughout the sequence preserves more of the vocabulary but lowers accuracy. They conclude that stability between consecutive updates is helpful, but strict long-term preservation is not.
Critical strengths
-
Well-defined problem. It identifies a failure mode specific to continual learning with discrete tokenizers: the same signal can receive a different code after an update. Standard continual-learning work, which focuses on continuous representations or parameters, does not address this.
-
Simple, cheap method. CAP-CL adds only a distillation term on the coarse codes, with no new trainable modules, and works on top of standard replay. It would be easy to adopt in other residual-VQ pipelines.
-
Careful ablations. The paper separates which layers are constrained (coarse only vs. all layers vs. fine only) from which teacher is used (previous stage vs. initial tokenizer). It also tests the layer choice with the total loss weight held fixed, which rules out a simple "more regularization" explanation.
-
Useful negative finding. Showing that stronger long-term code preservation lowers accuracy is informative. It challenges the assumption that interface stability is what matters.
-
Consistent results. The method wins in all five paired seeds against every baseline, with paired confidence intervals reported. The ranking also holds across three dataset orderings, two of which were built deliberately to be smooth and abrupt.
-
Heterogeneous benchmark. The stream of 21 public datasets varies widely in montage, sampling rate, number of classes and paradigm. Combined with the planned release of the registry, preprocessing pipeline and stage orders, it could be a useful resource for the community.
-
Transparent about limitations. The authors openly state the within-file split, the single architecture, and the fact that they cannot fully explain why the regularizer helps. Held-out datasets and hyperparameter-selection data are kept carefully separate from the evaluation data.
Critical weaknesses
-
Small gains. CAP-CL improves Final accuracy by about 2 points over replay alone and by only 1–2 points over the other baselines (1.1 over Feat-KD@Θ0, 1.8 over DER++). The confidence intervals capture only seed variance under a fixed split and order, not variability across subjects, splits or datasets, which in EEG is typically much larger.
-
Unexplained mechanism. The method is motivated by preserving code assignments, yet the paper shows that stronger preservation lowers accuracy. The authors admit they do not know why the regularizer helps, so the main claim rests on an empirical gain without a convincing explanation.
-
Motivation and protocol are at odds. The stated goal is a stable interface for downstream consumers, but the main evaluation retrains the readout after every update, which largely removes the drift problem. The frozen-consumer experiment, which directly tests the motivation, is presented only as a secondary diagnostic. In addition, most codes change anyway (L0 flip ≈ 0.88, L2 ≈ 0.99 relative to the initial tokenizer).
-
Little forgetting to fight. With replay and head rehearsal in a task-incremental setting, BWT is close to zero or positive for all replay-based methods. The benchmark therefore barely exercises the continual-learning problem the method is meant to solve.
-
Weak case for continual training. Joint retraining is very cheap at this scale (8,400 steps on a single GPU) and still performs best. The replay buffer grows to roughly 20k trials, a large fraction of the training data. The argument that sequential updates avoid costly retraining is not supported in this setting.
-
Very small update budget. Each stage uses 400 steps with batch size 8 (about 3,200 samples), so the tokenizer is only lightly perturbed. It is unclear whether the findings hold under realistic continual pretraining.
-
Missing baselines. There is no model trained from scratch per task (e.g., EEGNet or Riemannian/CSP methods) and no general-purpose time-series foundation model used as a frozen feature extractor. Without these, the value of continual tokenizer pretraining cannot be judged.
-
Unfair or mislabeled comparisons. The EEG foundation models (BIOT, LaBraM, CBraMod) show strongly negative BWT, which suggests they were fine-tuned without replay, yet they appear in the main table. "DER++" is a feature-MSE variant that does not replay logits. Only three methods (CAP-CL, Feat-KD@Θt−1, DER++) received hyperparameter tuning, while LwF, EWC and Feat-KD@Θ0 used fixed coefficients.
-
Limited generality. The study uses a single tokenizer architecture (the authors' reimplementation of BrainRVQ) and a single 64-channel preprocessing pipeline. The within-file splits share subjects and recordings between train and test.
-
Limited novelty. The method combines replay with distillation of the previous model's code assignments, applied to one layer. The closest related work on drift-aware continual tokenization (Feng et al., 2026) is cited but not compared against.
-
Clarity issues. The paper is dense with acronyms and nested controls, which obscures the main message. The method's acronym is expanded differently in the abstract and the introduction, and Appendix I has a title but no content.
Questions for authors
-
Mechanism. If stronger code preservation lowers accuracy, what do you think CAP-CL actually contributes? Could you provide an analysis that isolates the source of the gain, for example its effect on codebook utilization, feature geometry, or reconstruction quality on past domains?
-
Interface stability as the primary metric. Since the paper is motivated by stable codes for downstream consumers, why is the main evaluation based on a readout that is retrained after every stage? Could you report the frozen-consumer setting (Table 6) for all baselines and treat it as a primary result?
-
Absolute drift. Even with CAP-CL, L0 flip relative to the initial tokenizer is about 0.88 and L2 is about 0.99. In what sense is the interface preserved for a consumer that is not retrained?
-
Forgetting. BWT is near zero or positive for all replay-based methods. Can you show results in a regime where forgetting is substantial, such as a smaller buffer, no head rehearsal, or a class- or domain-incremental setting, to test whether CAP-CL helps where continual learning is actually difficult?
-
Baselines without pretraining. How does the pipeline compare with models trained from scratch per task (e.g., EEGNet, Riemannian or CSP-based methods) under the same splits? This is needed to judge whether continual tokenizer pretraining adds value.
-
General-purpose time-series models. How would a frozen general-purpose time-series foundation model (e.g., Chronos, MOMENT) with a linear probe perform on the same tasks?
-
Foundation-model baselines. How exactly were BIOT, LaBraM and CBraMod trained across the stream? Did they use replay and head rehearsal? If not, could you report them with the same continual-learning protocol so the comparison is fair?
-
Tuning parity. Why were LwF, EWC and Feat-KD@Θ0 not tuned with the same OFAT protocol as CAP-CL, Feat-KD@Θt−1 and DER++? Given margins of 1–2 points, would equal tuning change the ranking?
-
DER++ adaptation. The adapted DER++ uses feature MSE rather than logit replay. Could you include a version closer to the original, for example replaying stored code distributions or soft assignments?
-
Cost argument. Joint-Offline is cheap at this scale and still performs best, while the replay buffer grows to roughly 20k trials. Under what data or compute regime do you expect sequential updates to be clearly preferable? Can you show results with a fixed global memory budget?
-
Update budget. With 400 steps of batch size 8 per stage, the tokenizer is only lightly updated. Do the conclusions hold with substantially longer updates per stage?
-
Evaluation variance. The confidence intervals reflect seed variance only. Can you report variability across splits or, ideally, a subject-disjoint evaluation, since within-file splits share subjects and recordings?
-
Generality. Have you tested CAP-CL on other residual tokenizers (e.g., LaBraM-style VQ or audio codecs) to check that the coarse-only finding is not specific to your BrainRVQ reimplementation?
-
Related work. How does CAP-CL compare empirically with drift-aware continual tokenization (Feng et al., 2026) or codebook-refresh approaches adapted to this setting?
-
Missing content. Appendix I ("Seed-level robustness") is empty. Was content omitted? Also, which expansion of the CAP-CL acronym is correct?
2 comments
I thank the authors for the response. However, none of the additional analyses I requested were provided: no mechanism analysis (Q1), no regime with substantial forgetting (Q4), no non-pretrained or general time-series baselines (Q5–6), no re-run of the foundation models under the same CL protocol (Q7), no tuning parity (Q8), no fixed global memory budget or longer updates (Q10–11), no subject/session-disjoint evaluation (Q12), no additional tokenizer (Q13), and no empirical comparison with DACT (Q14).
More importantly, the response states that the quantitative frozen-consumer results (submitted Table 6) were removed because the diagnostic configuration fed waveform patches directly to the AR backbone, bypassing the tokenizer. This was the only experiment directly testing the paper’s motivation (a stable interface for consumers that are not retrained), and it was cited as a strength by several reviewers. Without it, the motivation rests only on a protocol where the readout is retrained after every stage, and the remaining evidence (initial-teacher control) indicates that stronger assignment preservation does not improve accuracy.
The paired improvement over ER is consistent across seeds, but it remains a ~2-point gain under a single order, a within-file split, and a single tokenizer, with an unexplained mechanism. I therefore maintain my score.
Response to Reviewer ypoF
Q1 · What does CAP-CL contribute? Table 2 reports Final BAcc of 0.421 for CAP-CL and 0.401 for ER. In the documented objective, setting the index-loss coefficient to zero recovers ER with the same replay and readout schedule. Table 4 further compares coarse-only and all-layer distillation at a common total coefficient, with Final BAcc of 0.421 and 0.405. These comparisons test the addition and layer allocation of the discrete loss. The design rationale is that the coarse assignment defines the partition on which residual codes are organized. Constraining that partition leaves fine layers trainable through reconstruction and quantization.
Table 3 compares teacher references with distillation on current and replay batches. The initial teacher preserves initial coarse assignments better than the previous-stage teacher (flip rates 0.568 and 0.876), while their Final BAcc values are 0.412 and 0.421. These values favor previous-stage supervision for adapted accuracy at the reported coefficient. Figure 3 shows a related pattern for replay-only L0 distillation, with lower adjacent-stage flips on capped replay (c,d) alongside substantial cumulative drift on held-out first-domain trials (a,b).
Q2–3 · Readout and token compatibility. The primary evaluation adapts the readout after each stage to measure the updated pipeline's Task-IL performance under a shared supervised budget. The previous-stage teacher is refreshed at stage boundaries, so the constraint follows the evolving model. Successive local constraints can accumulate changes relative to initialization, as Figure 3 illustrates. The revised manuscript omits the quantitative frozen-consumer results from submitted Table 6.
Q4 · Forgetting. In Task-IL with replay and head rehearsal, Table 2 reports near-zero or positive BWT. BWT measures each method's change from its own score immediately after learning a task. Final BAcc measures the accuracy reached across the stream. CAP-CL and ER report 0.421 and 0.401 Final BAcc with BWT of 0.011 and 0.004. These results describe accuracy under a retention-oriented protocol. Naive-CPT omits replay and head rehearsal and reports BWT of −0.041.
Q7 · EEG foundation-model references. Table 2 presents BIOT, LaBraM, and CBraMod as references with native Task-IL stacks. The available runner uses current-domain labels without replay or head rehearsal. The reference rows are interpreted in light of these protocol differences. The matched continual-pretraining comparisons share LexVQ and its documented replay and readout budgets.
Q8–9 · Coefficients and DER++ adaptation. Appendix F reports the recorded coefficients. LwF, EWC, and initial-teacher Feat-KD use fixed coefficients. The total-weight-matched block fixes the index-loss coefficient budget when varying constrained layers. Adapted DER++ stores waveforms and matches previous-stage pre-quantization features on replay. This instantiates continuous-feature retention during unlabeled tokenizer updates. The comparison with CAP-CL tests the specified feature and discrete assignment objectives at their reported settings.
Q10–11 · Update and data-access setting. Joint-Offline reaches 0.434 Final BAcc compared with 0.421 for CAP-CL. Both use 8,400 tokenizer updates overall, with 400 updates per streaming stage at batch size 8. Joint-Offline pools all 21 capped training subsets. Streaming uses the current domain and stored replay as each domain arrives. This addresses sequential data access. The buffer grows by domain and holds up to 20,480 prior trials by stage 21.
Q12–13 · Evaluation scope. Table 14 reports intervals computed from the five submitted seed-level differences for the primary order. The documented pairing protocol uses the same evaluation split within each seed. Main-21 uses trial-disjoint within-file splits, so subjects and recordings may be shared across training and evaluation. Yang2025 is the corpus-unseen probe, and Jeong2020MI is a same-study-family paradigm-transfer probe. The evaluation uses one residual tokenizer architecture and a shared 64-channel mapping.
Q14 · Tokenizer adaptation. DACT aligns soft first-layer assignments across stages and conditionally reassigns finer codes when the coarse code changes. CAP-CL distills hard coarse indices on current and replay EEG while training fine layers with the base tokenizer objective. The manuscript compares layer scope and teacher reference on a BrainRVQ-derived tokenizer. Feng et al., 2026, §3
Q15 · Appendix and naming. The revised PDF places Table 14 under Appendix I. We use Coarse Assignment Preservation for Continual Learning consistently.