- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces TRACE, an autoregressive framework designed to pre-train on EEG data by forecasting future patches. It utilizes a TR-MoE architecture that pairs spatial-temporal attention with a Cross-Channel Temporal Routing FFN (CTR-FFN) to assign channel-shared experts at every time step. Instead of forecasting a single subsequent patch, the pre-training phase leverages historical context to predict several future patches simultaneously. In their experimental results, across eight downstream datasets, TRACE outperformed baseline models on six of them.
Critical strengths
- The idea of temporal routing is novel in the EEG field.
- The downstream performance of the proposed method is promising.
- The ablation studies is well designed, particularly the controlled comparison showing that autoregressive pre-training consistently outperforms conventional masked reconstruction.
Critical weaknesses
- The justification for choosing forecasting over masked reconstruction as a pre-training objective is insufficiently explained. Relying exclusively on empirical gains is inadequate; the authors must provide stronger theoretical intuition and support this choice with relevant literature.
- The adoptation of causal attention is counterintuitive given the evaluation scope. While causal masking aligns with the proposed prediction task, it seems unnecessary for the downstream classification tasks evaluated in the paper, where the entire temporal sequence is already fully observable.
- The paper does not report GFLOPs across different channel counts and sequence lengths, making it unclear how the computational cost scales under heterogeneous EEG settings.
Questions for authors
- See the weaknesses.
- What is the reason behind sharing the temporal routing across all channels?
- Given the non-stationary characteristics of EEG signals, is it feasible to forecast them? Overall, I am currently not very convinced by the choice of the forecasting pretraining task; I expect more intuition and related work than just experimental results and authors’ response may affect my evaluation.
5 comments
Rebuttal Part 2
2. Causal attention and full-window downstream classification (W2)
Response: We retain causal attention to preserve the attention structure learned during autoregressive pre-training, and our direct fine-tuning comparison supports this choice. During pre-training, each token learns to represent its observed prefix. Switching to bidirectional attention during fine-tuning changes this conditioning structure. Although the complete input window permits bidirectional processing, our experiment shows that this change reduces average downstream performance from the same pre-trained checkpoint.
In addition, causal attention does not imply using only the final temporal token for classification. It constrains information flow within the encoder: each token attends to its available prefix. Our classification head uses the outputs from all channels and all temporal positions, which are flattened into a single vector and passed to a three-layer MLP. With encoder output , the prediction is
The encoder and classification head are fine-tuned end-to-end, allowing the final prediction to combine features from across the complete input window.
Specifically, we compared causal and bidirectional fine-tuning from the same autoregressively pre-trained checkpoint, keeping the downstream setup fixed and changing only the fine-tuning attention mode. The mean balanced accuracies are:
| Dataset | Causal FT (%) | Bidirectional FT (%) | Causal − bidirectional (pp) |
|---|---|---|---|
| FACED | 63.52 | 63.51 | +0.01 |
| ISRUC | 79.97 | 78.40 | +1.57 |
| BCIC2020-3 | 63.15 | 61.89 | +1.26 |
| CHB-MIT | 74.53 | 71.61 | +2.92 |
| PhysioNet-MI | 63.42 | 60.90 | +2.52 |
| Macro average | 68.92 | 67.26 | +1.66 |
Causal fine-tuning improves balanced accuracy on four datasets and performs essentially identically on FACED, giving a macro-average gain of 1.66 percentage points. FACED's Kappa and weighted F1 slightly favor bidirectional attention. The average covers the five datasets with completed comparisons.
We will clarify that classification uses all token representations, and add these fine-tuning results to the revised manuscript.
3. Why channels share temporal routing (Q2)
Response: We share expert selection across channels to adapt computation to the temporal context of a multichannel EEG observation. Our hypothesis is that a joint decision regularizes routing: channels observing the same temporal pattern can differ in amplitude and local noise, and independent decisions may respond to those differences. Sharing the decision encourages a common processing rule for that temporal pattern. Each expert still processes each channel's own representation, retaining channel-specific information.
Our routing-design ablation (, ) compares independent token routing with two channel-shared designs:
| Routing design | FACED BAcc (%) | PhysioNet-MI BAcc (%) | SEED-V BAcc (%) | Average (%) |
|---|---|---|---|---|
| Independent token routing (Vanilla MoE) | 62.39 | 62.76 | 38.43 | 54.53 |
| Channel-shared routing with mean-pooled context | 63.45 | 63.00 | 39.42 | 55.29 |
| Channel-shared routing with TemporalFormer (TRACE) | 63.52 | 63.42 | 40.32 | 55.75 |
Mean-pooled shared routing improves average balanced accuracy by 0.76 percentage points over token-wise routing, and TemporalFormer adds a further 0.46 points. These results support using a joint multichannel context to select experts for simultaneous channel observations.
To examine the role of history in this shared decision, we further restricted TemporalFormer to all channel tokens at the current time step. Restricting the router to current-step tokens reduces balanced accuracy by 3.83 percentage points on FACED and 1.09 points on ISRUC, with decreases in Kappa and weighted F1 as well. Since backbone attention is unchanged, the current-step representations can still contain temporal information. The comparison shows the additional benefit of giving TemporalFormer direct access to historical token representations when selecting experts.
| Dataset | TemporalFormer input | BAcc (%) ↑ | Kappa ↑ | Weighted F1 (%) ↑ |
|---|---|---|---|---|
| FACED | History through the current step (TRACE) | 63.52 | 0.5883 | 63.90 |
| FACED | Current step only | 59.69 | 0.5448 | 60.00 |
| ISRUC | History through the current step (TRACE) | 79.97 | 0.7711 | 82.09 |
| ISRUC | Current step only | 78.88 | 0.7584 | 80.94 |
Together, these results motivate our design: share expert selection across channels to coordinate the processing of simultaneous observations, and condition that selection on history so it can adapt as the recording evolves.
Rebuttal Part 3
4. Computational scaling with channels and sequence length (W3)
Response: Thank you for suggesting a broader analysis of computational cost. We profiled TRACE across 20 input configurations, covering 16–128 channels and 1–30 seconds, to quantify how computation scales with heterogeneous EEG inputs. The model configuration is fixed at 64 experts, top-8 activation, and routed expert width 200, using the MegaBlocks implementation.
Let denote the channel count and the number of temporal patches per channel. Each patch contains 200 samples (one second at 200 Hz), giving input tokens. We report backbone forward GFLOPs per sample for convolution, linear, attention, and expert operations, counting each multiply–accumulate as two FLOPs:
| Channels | (1 s) | (4 s) | (10 s) | (30 s) |
|---|---|---|---|---|
| 16 | 0.617 | 2.469 | 6.182 | 18.637 |
| 19 | 0.726 | 2.905 | 7.273 | 21.930 |
| 32 | 1.200 | 4.803 | 12.025 | 36.260 |
| 64 | 2.380 | 9.529 | 23.860 | 71.948 |
| 128 | 4.801 | 19.218 | 48.119 | 145.094 |
Across the evaluated range, computation grows approximately in proportion to channel count and sequence length. At 32 channels, extending the input from 4 to 30 seconds (7.5×) increases computation from 4.803 to 36.260 GFLOPs (7.55×). At 30 seconds, increasing the channel count from 16 to 128 (8×) increases computation from 18.637 to 145.094 GFLOPs (7.79×).
At fixed model dimensions, token-wise projections and expert computation scale linearly with , while spatial and temporal attention introduce terms of , where is the hidden dimension. The near-proportional growth in the table therefore characterizes the tested input range. We will add this table and the detailed profiling procedure to the revised appendix.
Rebuttal Part 1
Thank you for your constructive feedback. We clarify the motivation and implementation below and provide an additional comparison of causal and bidirectional fine-tuning.
1. Forecasting motivation and non-stationary EEG (W1, Q3)
Response: Our motivation is to learn EEG representations that capture the temporal evolution of brain activity. Evidence from MEG and intracranial recordings suggests that brain activity exhibits structured temporal dynamics, including transient networks with distinct spectral and coupling patterns [1] and transitions between behavior-associated states [2]. This motivates a predictive objective that encourages the model to retain information about how the observed activity may evolve.
We implement this objective through forecasting: predicting subsequent EEG activity from the observed history. Whereas bidirectional masked reconstruction can use observations on both sides of a missing segment, forecasting requires the representation of the preceding context to support prediction of what comes next.
The intuition is that two histories requiring different optimal future predictions should remain distinguishable in the learned representation; otherwise, the prediction head must produce the same output for both. TRACE applies this principle across multiple horizons by predicting the next 1, 2, and 4 patches. We hypothesize that the resulting sensitivity to temporal evolution benefits downstream EEG recognition.
This motivation also accommodates EEG non-stationarity. As brain activity changes, its recent history can still provide information about what follows. TRACE conditions each prediction on the observed prefix, while the Huber loss reduces the influence of large residuals arising from abrupt transitions and artifacts.
Recent precedents include TimesFM-3 (2026) [3], which combines causal temporal attention with cross-variate attention for zero-shot multivariate forecasting; BrainGPT (2025 revision) [4], which uses next-signal pre-training for downstream EEG tasks; and Chronos-2 (2025) [5], which demonstrates multivariate and covariate-informed forecasting, including energy and retail applications.
Our matched TRACE comparison supports this choice: forecasting improves balanced accuracy on four of five datasets, increasing the macro average from 58.56% to 64.59%, with PhysioNet-MI the exception. The backbone architecture, corpus, optimization budget, and downstream protocol are fixed, while attention constraints and input corruption follow each formulation.
[1] Vidaurre et al. Spontaneous cortical activity transiently organises into frequency specific phase-coupling networks. Nature Communications, 2018. Paper.
[2] Wang et al. A week in the life of the human brain reveals stable states punctuated by chaotic-like transitions. Nature Communications, 2026. Paper.
[3] Jain and Sen. TimesFM-3: A zero-shot foundation model for multivariate forecasting. Google Research, August 31, 2026. Official technical blog.
[4] Yue et al. BrainGPT: Unleashing the Potential of EEG Generalist Foundation Model by Autoregressive Pre-training. arXiv:2410.19779v2, revised August 29, 2025. Preprint.
[5] Ansari et al. Chronos-2: From Univariate to Universal Forecasting. arXiv:2510.15821, 2025. Technical report.
[6] Zheng and Lu. A multimodal approach to estimating vigilance using EEG and forehead EOG. Journal of Neural Engineering, 2017. Paper. SEED-VIG dataset.
[7] Wang et al. CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding. ICLR, 2025. Table 11.