acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Temporal Routing with Autoregressive Cross-channel Experts for EEG Representation Learning

Abstract

Learning transferable representations for electroencephalography (EEG) remains challenging because EEG signals are inherently multi-channel and non-stationary. Channels observed at the same time provide coupled measurements of neural activity, while the relevant temporal dynamics vary across contexts. This structure is poorly matched by architectures that apply uniform computation across time or route each channel patch independently. To this end, we propose TRACE, an autoregressive EEG pre-training framework that predicts future EEG patches from observed history while performing temporally adaptive and cross-channel coherent computation. At each temporal step, TRACE derives an expert routing decision from the cross-channel history and applies it jointly to all channels at that step. This preserves instantaneous cross-channel coherence while allowing different temporal regimes to activate different computation. Since routing is defined over the available channel set and temporal history, TRACE is compatible with heterogeneous pre-training across corpora with different channel counts, montages, sequence lengths, and recording domains. Across eight downstream EEG benchmarks, TRACE is evaluated in both settings: when downstream domains are seen only as unlabeled pre-training data and when downstream datasets are completely unseen during pre-training. It achieves the best mean performance on six of the eight benchmarks and remains competitive on the two motor imagery tasks, with ablations supporting the importance of cross-channel temporal routing.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG, representation learning, self-supervised pretraining, mixture-of-experts, autoregressive modeling, cross-channel routing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZzdKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TRACE, an autoregressive framework designed to pre-train on EEG data by forecasting future patches. It utilizes a TR-MoE architecture that pairs spatial-temporal attention with a Cross-Channel Temporal Routing FFN (CTR-FFN) to assign channel-shared experts at every time step. Instead of forecasting a single subsequent patch, the pre-training phase leverages historical context to predict several future patches simultaneously. In their experimental results, across eight downstream datasets, TRACE outperformed baseline models on six of them.

Critical strengths
  • The idea of temporal routing is novel in the EEG field.
  • The downstream performance of the proposed method is promising.
  • The ablation studies is well designed, particularly the controlled comparison showing that autoregressive pre-training consistently outperforms conventional masked reconstruction.
Critical weaknesses
  • The justification for choosing forecasting over masked reconstruction as a pre-training objective is insufficiently explained. Relying exclusively on empirical gains is inadequate; the authors must provide stronger theoretical intuition and support this choice with relevant literature.
  • The adoptation of causal attention is counterintuitive given the evaluation scope. While causal masking aligns with the proposed prediction task, it seems unnecessary for the downstream classification tasks evaluated in the paper, where the entire temporal sequence is already fully observable.
  • The paper does not report GFLOPs across different channel counts and sequence lengths, making it unclear how the computational cost scales under heterogeneous EEG settings.
Questions for authors
  • See the weaknesses.
  • What is the reason behind sharing the temporal routing across all channels?
  • Given the non-stationary characteristics of EEG signals, is it feasible to forecast them? Overall, I am currently not very convinced by the choice of the forecasting pretraining task; I expect more intuition and related work than just experimental results and authors’ response may affect my evaluation.
5 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Thank you for your constructive feedback. We clarify the motivation and implementation below and provide an additional comparison of causal and bidirectional fine-tuning.

1. Forecasting motivation and non-stationary EEG (W1, Q3)

Response: Our motivation is to learn EEG representations that capture the temporal evolution of brain activity. Evidence from MEG and intracranial recordings suggests that brain activity exhibits structured temporal dynamics, including transient networks with distinct spectral and coupling patterns [1] and transitions between behavior-associated states [2]. This motivates a predictive objective that encourages the model to retain information about how the observed activity may evolve.

We implement this objective through forecasting: predicting subsequent EEG activity from the observed history. Whereas bidirectional masked reconstruction can use observations on both sides of a missing segment, forecasting requires the representation of the preceding context to support prediction of what comes next.

The intuition is that two histories requiring different optimal future predictions should remain distinguishable in the learned representation; otherwise, the prediction head must produce the same output for both. TRACE applies this principle across multiple horizons by predicting the next 1, 2, and 4 patches. We hypothesize that the resulting sensitivity to temporal evolution benefits downstream EEG recognition.

This motivation also accommodates EEG non-stationarity. As brain activity changes, its recent history can still provide information about what follows. TRACE conditions each prediction on the observed prefix, while the Huber loss reduces the influence of large residuals arising from abrupt transitions and artifacts.

Recent precedents include TimesFM-3 (2026) [3], which combines causal temporal attention with cross-variate attention for zero-shot multivariate forecasting; BrainGPT (2025 revision) [4], which uses next-signal pre-training for downstream EEG tasks; and Chronos-2 (2025) [5], which demonstrates multivariate and covariate-informed forecasting, including energy and retail applications.

Our matched TRACE comparison supports this choice: forecasting improves balanced accuracy on four of five datasets, increasing the macro average from 58.56% to 64.59%, with PhysioNet-MI the exception. The backbone architecture, corpus, optimization budget, and downstream protocol are fixed, while attention constraints and input corruption follow each formulation.

[1] Vidaurre et al. Spontaneous cortical activity transiently organises into frequency specific phase-coupling networks. Nature Communications, 2018. Paper.

[2] Wang et al. A week in the life of the human brain reveals stable states punctuated by chaotic-like transitions. Nature Communications, 2026. Paper.

[3] Jain and Sen. TimesFM-3: A zero-shot foundation model for multivariate forecasting. Google Research, August 31, 2026. Official technical blog.

[4] Yue et al. BrainGPT: Unleashing the Potential of EEG Generalist Foundation Model by Autoregressive Pre-training. arXiv:2410.19779v2, revised August 29, 2025. Preprint.

[5] Ansari et al. Chronos-2: From Univariate to Universal Forecasting. arXiv:2510.15821, 2025. Technical report.

[6] Zheng and Lu. A multimodal approach to estimating vigilance using EEG and forehead EOG. Journal of Neural Engineering, 2017. Paper. SEED-VIG dataset.

[7] Wang et al. CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding. ICLR, 2025. Table 11.

AuthorsOpenReview ↗

Rebuttal Part 2

2. Causal attention and full-window downstream classification (W2)

Response: We retain causal attention to preserve the attention structure learned during autoregressive pre-training, and our direct fine-tuning comparison supports this choice. During pre-training, each token learns to represent its observed prefix. Switching to bidirectional attention during fine-tuning changes this conditioning structure. Although the complete input window permits bidirectional processing, our experiment shows that this change reduces average downstream performance from the same pre-trained checkpoint.

In addition, causal attention does not imply using only the final temporal token for classification. It constrains information flow within the encoder: each token attends to its available prefix. Our classification head uses the outputs from all channels and all temporal positions, which are flattened into a single vector and passed to a three-layer MLP. With encoder output HL∈RC×n×dH^L \in \mathbb{R}^{C \times n \times d}, the prediction is

y^=MLP⁡(vec⁡(HL)).\hat{y}=\operatorname{MLP}\left(\operatorname{vec}(H^L)\right).

The encoder and classification head are fine-tuned end-to-end, allowing the final prediction to combine features from across the complete input window.

Specifically, we compared causal and bidirectional fine-tuning from the same autoregressively pre-trained checkpoint, keeping the downstream setup fixed and changing only the fine-tuning attention mode. The mean balanced accuracies are:

DatasetCausal FT (%)Bidirectional FT (%)Causal − bidirectional (pp)
FACED63.5263.51+0.01
ISRUC79.9778.40+1.57
BCIC2020-363.1561.89+1.26
CHB-MIT74.5371.61+2.92
PhysioNet-MI63.4260.90+2.52
Macro average68.9267.26+1.66

Causal fine-tuning improves balanced accuracy on four datasets and performs essentially identically on FACED, giving a macro-average gain of 1.66 percentage points. FACED's Kappa and weighted F1 slightly favor bidirectional attention. The average covers the five datasets with completed comparisons.

We will clarify that classification uses all token representations, and add these fine-tuning results to the revised manuscript.

3. Why channels share temporal routing (Q2)

Response: We share expert selection across channels to adapt computation to the temporal context of a multichannel EEG observation. Our hypothesis is that a joint decision regularizes routing: channels observing the same temporal pattern can differ in amplitude and local noise, and independent decisions may respond to those differences. Sharing the decision encourages a common processing rule for that temporal pattern. Each expert still processes each channel's own representation, retaining channel-specific information.

Our routing-design ablation (N=8N=8, K=2K=2) compares independent token routing with two channel-shared designs:

Routing designFACED BAcc (%)PhysioNet-MI BAcc (%)SEED-V BAcc (%)Average (%)
Independent token routing (Vanilla MoE)62.3962.7638.4354.53
Channel-shared routing with mean-pooled context63.4563.0039.4255.29
Channel-shared routing with TemporalFormer (TRACE)63.5263.4240.3255.75

Mean-pooled shared routing improves average balanced accuracy by 0.76 percentage points over token-wise routing, and TemporalFormer adds a further 0.46 points. These results support using a joint multichannel context to select experts for simultaneous channel observations.

To examine the role of history in this shared decision, we further restricted TemporalFormer to all channel tokens at the current time step. Restricting the router to current-step tokens reduces balanced accuracy by 3.83 percentage points on FACED and 1.09 points on ISRUC, with decreases in Kappa and weighted F1 as well. Since backbone attention is unchanged, the current-step representations can still contain temporal information. The comparison shows the additional benefit of giving TemporalFormer direct access to historical token representations when selecting experts.

DatasetTemporalFormer inputBAcc (%) ↑Kappa ↑Weighted F1 (%) ↑
FACEDHistory through the current step (TRACE)63.520.588363.90
FACEDCurrent step only59.690.544860.00
ISRUCHistory through the current step (TRACE)79.970.771182.09
ISRUCCurrent step only78.880.758480.94

Together, these results motivate our design: share expert selection across channels to coordinate the processing of simultaneous observations, and condition that selection on history so it can adapt as the recording evolves.

AuthorsOpenReview ↗

Rebuttal Part 3

4. Computational scaling with channels and sequence length (W3)

Response: Thank you for suggesting a broader analysis of computational cost. We profiled TRACE across 20 input configurations, covering 16–128 channels and 1–30 seconds, to quantify how computation scales with heterogeneous EEG inputs. The model configuration is fixed at 64 experts, top-8 activation, and routed expert width 200, using the MegaBlocks implementation.

Let CC denote the channel count and nn the number of temporal patches per channel. Each patch contains 200 samples (one second at 200 Hz), giving CnCn input tokens. We report backbone forward GFLOPs per sample for convolution, linear, attention, and expert operations, counting each multiply–accumulate as two FLOPs:

Channels CCn=1n=1 (1 s)n=4n=4 (4 s)n=10n=10 (10 s)n=30n=30 (30 s)
160.6172.4696.18218.637
190.7262.9057.27321.930
321.2004.80312.02536.260
642.3809.52923.86071.948
1284.80119.21848.119145.094

Across the evaluated range, computation grows approximately in proportion to channel count and sequence length. At 32 channels, extending the input from 4 to 30 seconds (7.5×) increases computation from 4.803 to 36.260 GFLOPs (7.55×). At 30 seconds, increasing the channel count from 16 to 128 (8×) increases computation from 18.637 to 145.094 GFLOPs (7.79×).

At fixed model dimensions, token-wise projections and expert computation scale linearly with CnCn, while spatial and temporal attention introduce terms of O(nC2d+Cn2d)O(nC^2d + Cn^2d), where dd is the hidden dimension. The near-proportional growth in the table therefore characterizes the tested input range. We will add this table and the detailed profiling procedure to the revised appendix.

Reviewer_KcCuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRACE, an autoregressive EEG foundation model that introduces temporal routing with cross-channel shared experts. TemporalFormer summarizes the multi-channel history to select a common set of experts for all channels at each time step. The model is pretrained through multi-horizon forecasting and evaluated on eight downstream EEG datasets, with additional ablations examining its routing design and pretraining objectives.

Critical strengths

S1. A focused architectural contribution

The paper explores the granularity of expert routing in multi-channel EEG and proposes a design that shares expert selection across channels while allowing routing decisions to vary over time. This offers a specific architectural alternative to conventional token-wise MoE routing.

S2. Substantial experimental evaluation

The paper evaluates TRACE across eight downstream datasets and includes ablations on routing strategies, pretraining objectives, expert configurations, and data composition. These experiments provide useful evidence for assessing the proposed approach.

Critical weaknesses

W1. Insufficient discussion of closely related work

The paper lacks adequate discussion and comparison with EEGMoE and Uni-NTFM, which are highly relevant to its MoE architecture and expert specialization design. Recent EEG foundation models, including EEGPT (Wang et al.), LUNA, and REVE, also deserve more attention to better position TRACE within existing research.

W2. Limited justification for cross-channel shared routing

The argument that shared expert selection preserves cross-channel coherence lacks sufficient theoretical or empirical support. Assigning identical experts to all channels does not necessarily establish meaningful physiological coherence. The paper also provides limited discussion of when this inductive bias is beneficial or restrictive, while the relationship between expert switching and EEG dynamics remains unclear.

W3. Unclear relationship between channel-order dependence and montage compatibility

MS-CCE applies convolutions along a fixed, dataset-specific channel order, while the paper emphasizes compatibility with heterogeneous electrode montages. The implications of this order dependence for cross-montage generalization are not sufficiently explained or evaluated.

W4. Limited evidence for temporal adaptation on SEED-V

TRACE uses 1-second patches, while SEED-V also uses 1-second input windows. Each sample therefore contains only one temporal patch, preventing within-sample changes in expert selection. Improvements on SEED-V cannot directly demonstrate the effectiveness of temporally adaptive routing.

W5. Limited improvement over simpler temporal routing

In Table 15, TRACE improves average Balanced Accuracy over Mean Temporal Routing by only 0.46 percentage points (55.75% versus 55.29%), despite requiring additional computation. The incremental benefit of the more complex TemporalFormer-based routing mechanism appears modest.

W6. Incomplete reporting of the final model's computational cost

Appendix H.11 reports parameter counts and GFLOPs for the reduced (N=8, K=2) configuration. However, corresponding total parameters, activated parameters, and computational costs are not reported for the final (N=64, K=8) model, making its efficiency trade-offs difficult to assess.

Questions for authors

Q1. Could the authors isolate the benefit of cross-channel shared routing by comparing it with independent channel routing under the same TemporalFormer context and computational budget?

Q2. Appendix H.10 shows time-varying expert selection, but does historical context actually drive these changes and improve performance? Could the authors provide a history-ablated routing comparison?

Q3. Could the authors provide evidence of cross-montage generalization despite MS-CCE's dependence on fixed channel ordering, rather than merely demonstrating support for different channel counts?

5 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Thank you for your detailed review and constructive suggestions. We clarify the scope of our claims and report additional experiments on independent channel routing, historical context, and channel-order sensitivity below.

1. Positioning TRACE relative to related work (W1)

Response: Thank you for highlighting these relevant works. EEGMoE and Uni-NTFM introduce expert specialization through token-level routing. EEGPT (Wang et al.), LUNA, and REVE develop masked EEG representation learning, with advances in representation alignment and the handling of heterogeneous channel configurations. These works establish relevant foundations for expert-based modeling and learning across diverse EEG datasets.

TRACE's specific contribution is to combine history-conditioned, channel-shared expert selection with multi-horizon autoregressive forecasting. At each layer and temporal step, TemporalFormer summarizes the multichannel context up to and including the current step, and the router uses this summary to select a common expert mixture for all channels. Each selected expert processes the channels' individual representations. This selection can change as the recording evolves, coordinating computation across simultaneous observations while adapting it over time. The forecasting objective trains these representations to predict future signal patches over multiple horizons from the observed prefix. We will expand the revised related-work section to acknowledge these connections and clarify this distinction in routing granularity and pre-training objective.

2. Justification for shared routing and cross-channel coherence (W2)

Response: Thank you for asking us to clarify the rationale. Our hypothesis is that a joint decision regularizes routing: channels observing the same temporal pattern can differ in amplitude and local noise, and independent routing decisions may respond to these differences. Selecting experts from their joint context encourages a common processing rule for that pattern. Each selected expert still processes each channel's own representation, retaining channel-specific information.

TemporalFormer implements this principle by summarizing the available multichannel history at each layer and step; the router uses this summary to select an expert mixture shared across the simultaneous channel observations. As the context evolves, the selection can change. The design therefore combines coordination across channels with adaptation across time.

Our additional experiments support both aspects. For cross-channel coordination, the comparison in Q1 equips independent routing with TemporalFormer context, yet TRACE achieves higher mean balanced accuracy by 0.91 and 0.50 percentage points on FACED and PhysioNet-MI, respectively. For the contribution of history, the ablation in Q2 restricts the router's direct input to the current step while retaining temporal processing in the main backbone; balanced accuracy decreases by 3.83 points on FACED and 1.09 points on ISRUC. Together, these results support sharing expert selection across channels and using historical context to guide that selection.

This inductive bias is intended for patterns whose processing benefits from joint multichannel context. It may be restrictive for focal events confined to a few electrodes, where channel-specific routing could offer greater flexibility. We will clarify that “cross-channel coherence” describes coordinated expert selection. The time-varying selections in Appendix H.10 illustrate adaptation across steps; they do not establish a correspondence between individual experts and physiological states.

AuthorsOpenReview ↗

Rebuttal Part 2

3. Shared versus independent channel routing with TemporalFormer context (Q1)

Response: Thank you for this suggestion. We extend the routing ablations in Appendix Table 15 with a TemporalFormer + Independent Routing baseline. This baseline incorporates TemporalFormer context into each channel's routing decision while allowing channels to select their top-KK experts independently. TRACE instead uses a common expert selection across channels at each temporal step.

We will report the full results, including balanced accuracy, Kappa, and weighted F1 with their means and standard deviations, in the revised appendix.

We also include Mean Temporal Routing, which mean-pools channel embeddings at each temporal step to obtain a common routing context and shares the resulting expert selection across channels. This provides a simpler shared-routing reference alongside the TemporalFormer-based variants. The balanced-accuracy results are:

Routing designFACED (%)PhysioNet-MI (%)Two-dataset average (%)
Vanilla MoE: independent token routing62.39 ± 1.0462.76 ± 0.7062.58
TemporalFormer + independent routing62.61 ± 0.6662.92 ± 0.6462.77
Mean Temporal Routing: mean pooling + shared selection63.45 ± 0.5363.00 ± 0.8463.23
TRACE: TemporalFormer + shared selection63.52 ± 0.3763.42 ± 0.9063.47

Across these two datasets, mean-pooled shared routing improves average balanced accuracy over independent token routing by 0.65 percentage points, supporting the use of a joint channel context for shared expert selection. Within the shared-routing design, replacing mean pooling with TemporalFormer further increases average balanced accuracy by approximately 0.25 points. Differences are computed before rounding the displayed averages.

The additional TemporalFormer-equipped independent-routing baseline reaches 62.77% on average, while TRACE is higher by 0.91 and 0.50 points on FACED and PhysioNet-MI, respectively. Together, these comparisons support combining joint-context aggregation with shared expert selection; TRACE achieves the highest mean balanced accuracy on both datasets.

4. Contribution of historical context to routing (Q2)

Response: Thank you for this suggestion. We introduced a current-step-only routing variant to examine the contribution of temporal context to shared expert selection. A weight-sharing branch processes each time step independently from the raw EEG input, using all channels at that step to compute TemporalFormer-based routing decisions at every layer. The resulting expert selections and mixture weights are then applied in the normal full-sequence backbone.

Channel-shared expert selection and the backbone's causal attention are retained. The current-step-only constraint is applied during both pre-training and fine-tuning, using the same downstream evaluation protocol.

DatasetRouting contextBalanced accuracy (%)KappaWeighted F1 (%)
FACEDFull context63.52 ± 0.370.5883 ± 0.004263.90 ± 0.43
FACEDCurrent step only59.69 ± 0.920.5448 ± 0.010360.00 ± 0.97
ISRUCFull context79.97 ± 0.130.7711 ± 0.004882.09 ± 0.40
ISRUCCurrent step only78.88 ± 0.890.7584 ± 0.008580.94 ± 0.99

Restricting routing to current-step observations reduces balanced accuracy by 3.83 percentage points on FACED and 1.09 points on ISRUC, with corresponding decreases in Kappa and weighted F1.

Even when the main backbone retains temporal processing, selecting experts from current-step observations yields lower performance. This supports the contribution of historical context to expert selection and complements the routing visualization with a downstream ablation. It does not, by itself, identify the physiological meaning of individual routing transitions.

AuthorsOpenReview ↗

Rebuttal Part 3

5. Channel-order dependence and cross-montage generalization (W3, Q3)

Response: Thank you for asking us to clarify the role of channel ordering. Our claim is that different datasets can retain their own channel sets and montage configurations without being aligned to a common electrode set or a single global channel order. Within each dataset, we use a consistent channel order. These are compatible requirements: the model can process heterogeneous channel configurations across datasets while remaining sensitive to channel ordering within a dataset.

MS-CCE applies the same multi-scale depth-wise convolutional filters along the available channel axis, without requiring an electrode-specific embedding table. TemporalFormer then summarizes the available multichannel context into a routing vector. This allows the architecture to operate across dataset-specific channel sets. Because MS-CCE uses neighborhoods defined by the input order, shuffling channels within a dataset changes the context seen by its filters and can affect performance.

We evaluated two independently sampled channel permutations per dataset, each fixed across training, validation, and test inputs. Only channel order changed; all variants were fine-tuned from the same pre-trained checkpoint using the same protocol. Results are mean ± standard deviation over five seeds.

DatasetChannel orderBalanced accuracy (%)KappaWeighted F1 (%)
FACEDOriginal63.52 ± 0.370.5883 ± 0.004263.90 ± 0.43
FACEDShuffle 161.43 ± 0.280.5648 ± 0.003161.77 ± 0.21
FACEDShuffle 261.59 ± 0.800.5664 ± 0.009862.03 ± 0.79
PhysioNet-MIOriginal63.42 ± 0.900.5122 ± 0.012063.65 ± 0.96
PhysioNet-MIShuffle 161.29 ± 1.040.4838 ± 0.014061.47 ± 1.13
PhysioNet-MIShuffle 260.29 ± 0.840.4705 ± 0.011260.62 ± 0.89

Both permutations lower performance, with balanced-accuracy drops of 1.93–2.09 points on FACED and 2.13–3.13 points on PhysioNet-MI, indicating modest but consistent sensitivity to channel order.

This sensitivity is consistent with our use of a fixed order within each dataset and does not imply that different datasets must share an identical channel configuration. We will clarify the scope of our claim as supporting heterogeneous datasets without global channel alignment, while retaining a consistent channel order within each dataset.

6. Interpretation of the SEED-V results (W4)

Response: We agree. In this setting, each SEED-V sample contains a single temporal patch. The routing decision can depend on the multichannel content of that patch and differ across samples, but it cannot switch across temporal steps within the same sample. We will revise the interpretation accordingly.

Reviewer_RsaHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRACE, an autoregressive EEG foundation model that forecasts future raw patches while routing all channels at a given time step through a shared subset of experts. Each block combines factorized causal spatial–temporal attention with a TemporalFormer module that summarizes the observed cross-channel history and selects specialized experts, together with an always-active shared expert. TRACE is pretrained on six heterogeneous EEG sources with 16–128 channels and evaluated on eight downstream datasets under both seen-domain and unseen-dataset transfer settings. The reported results lead on six datasets by mean performance, and the paper includes ablations of the pretraining objective, routing structure, data composition, forecasting horizons, expert scaling, patch length, and embedding design.

Critical strengths
  1. The routing problem is clearly motivated. Sharing a routing decision across simultaneous channels is a plausible compromise between independent token routing, which may fragment a multi-channel observation, and recording-level routing, which cannot adapt over time. The causal formulation is also appropriate for learning predictive temporal representations.

  2. The evaluation spans diverse tasks and transfer regimes. The eight benchmarks cover emotion, motor imagery, sleep, seizure, imagined speech, and clinical event classification, and most use subject-wise or official splits. Reporting five-seed results and separating downstream datasets that were included only as unlabeled pretraining sources from entirely unseen datasets are both useful choices.

  3. The appendix contains a broad set of ablations. In particular, the matched autoregressive-versus-masked comparison, source-composition study, shared-expert removal, horizon sweep, embedding ablations, routing diagnostics, and explicit computational profile help expose both benefits and limitations of the design.

Critical weaknesses
  1. The state-of-the-art comparison is not controlled. Baseline numbers are taken from the respective publications rather than obtained under a shared preprocessing, split, fine-tuning, and hyperparameter-selection pipeline, and no statistical comparison is performed. The final TRACE model uses 64 specialized experts per layer with eight active experts, yet its total parameters, activated parameters, training cost, and inference cost are not reported alongside the much smaller baselines. Numerical ranking across these heterogeneous protocols is therefore insufficient evidence for a general performance advantage.

  2. The central routing ablation is both configuration- and compute-confounded. The routing experiments use a reduced 8-expert/top-2 model, whereas the reported final model uses 64 experts with top-8 activation. Even in the reduced setting, TRACE adds TemporalFormer and uses 24.72 GFLOPs per sample, 36.7% more than the dense baseline and substantially more than the token-wise MoE. The approximately one-point average gain therefore does not isolate cross-channel shared routing from additional active parameters and computation, and it is unclear whether the conclusion transfers to the final configuration.

Questions for authors

See Weaknesses.

4 comments

Thank you for your feedback. We clarify the evaluation protocols, statistical scope, and computational costs below, and distinguish the evidence for routing design from that for model scaling.

1. Benchmark fairness, statistical comparisons, and computational cost (W1)

Response: We respectfully clarify that using published baseline results follows established practice in EEG foundation-model evaluation. LaBraM draws its baseline results from the BIOT study; CBraMod reuses previously reported results where available; and REVE explicitly reports baseline results from existing studies. These provide direct precedents for our comparison strategy. See LaBraM, §3.2, CBraMod, §3.2, and REVE, §4.

These benchmark comparisons also accommodate different model sizes. For example, LaBraM compares its 5.8M, 46M, and 369M variants against the 3.2M-parameter BIOT in the same performance tables, with model sizes explicitly disclosed. LaBraM, Tables 1–2. This supports comparing complete methods at their respective model sizes while reporting capacity and computational cost. Following this practice, our benchmark comparison uses established evaluation protocols CBraMod, §3.3. We therefore consider this a fair comparison of reported benchmark performance. Comparable task definitions, data partitions, and evaluation metrics provide the basis for this comparison, while method-specific processing and optimization choices should be documented rather than required to be identical. Our internal ablations separately examine the contributions of TRACE's individual components.

We have also supplemented the five-seed mean ± standard deviation results with statistical comparisons of the internal routing ablations using five common seeds. These include paired confidence intervals, exact two-sided sign-flip tests, and Holm correction across nine comparisons.

To make the model-capacity and computation differences explicit, we provide the following profile for the measured 64-expert, top-8 MegaBlocks configuration:

MetricValue
Total parameters, including forecasting heads103.895 M
Active parameters, including forecasting heads23.255 M
Backbone forward computation21.930 GFLOPs/sample
Forward computation including three forecasting heads22.249 GFLOPs/sample

Costs use a 19-channel, 30-second input at 200 Hz and count the main matrix and convolution operations, excluding the downstream classifier. Pre-training takes 40 hours on four H100 GPUs (160 GPU-hours). We will include these costs and the detailed counting convention in the revised appendix.

2. Routing benefits and applicability to the final configuration (W2)

Response: We use the smaller N=8N=8, K=2K=2 configuration to compare routing designs under a common experimental setting while keeping the ablation study computationally manageable. This is an appropriate setting for testing the routing mechanism, with expert scaling examined separately. The Mean Temporal Routing control provides evidence that shared temporal routing contributes beyond additional model capacity or computation. In the N=8N=8, K=2K=2 ablation, it has the same total and active parameter counts and nearly identical computation as independent token routing, while achieving higher reported balanced accuracy on both datasets:

Routing designTotal params. (M)Active params. (M)GFLOPs/sampleFACED BAcc (%)PhysioNet-MI BAcc (%)
Independent token routing (Vanilla MoE)33.1615.8818.35262.3962.76
Mean Temporal Routing33.1615.8818.33163.4563.00
TemporalFormer routing (TRACE)38.9621.6820.59963.5263.42

The table shows the original ablation means; updated costs use the same input and include forecasting heads. Both temporal-routing variants improve on token routing, supporting joint channel context and shared expert selection. TemporalFormer achieves higher means with additional computation.

Our expert-scaling experiments complement these routing ablations by varying N∈{8,16,32,64}N\in\{8,16,32,64\} at K=2K=2 and K∈{1,2,4,8}K\in\{1,2,4,8\} at N=16N=16, while keeping the activated expert width—and hence the dominant expert computation—fixed. The N=64N=64 and K=8K=8 settings achieve the highest mean performance in their respective sweeps. Together, the smaller-scale ablations assess the routing design, while the scaling experiments support the use of larger expert pools and more active experts under a controlled computation budget.

AuthorsOpenReview ↗

Discussion

Dear Reviewer RsaH,

Thank you again for your time and thoughtful feedback. We have posted our responses, including additional experiments and clarifications addressing your concerns. We would appreciate your thoughts on whether these additions resolve the issues you raised, and we are happy to clarify any remaining questions.

Best regards,

The Authors

Thanks for the additional experiments. However, my concerns about benchmark fairness and whether the routing gains justify the added complexity remain. I will keep my original rating.

Thank you for your follow-up. We would like to address your remaining concerns about benchmark fairness and the benefit of the added routing complexity.

Benchmark fairness. Our main table compares complete methods using published results under established benchmark protocols, following the practice of LaBraM, CBraMod, and REVE discussed in our first response. Such a comparison does not require identical model sizes or method-specific training recipes. Its purpose is to assess the performance of the methods as developed, with their evaluation settings and computational costs made explicit. The benchmark comparison evaluates TRACE as a complete framework, while the internal ablations examine which routing choices contribute to its performance. To help us address this concern more concretely, could you specify which baseline comparisons and evaluation settings you consider insufficiently comparable, and what additional controlled comparisons would resolve your concern?

Routing benefits and complexity. Mean Temporal Routing improves on independent token routing on FACED and PhysioNet-MI with the same parameter counts and nearly identical computation. This provides evidence that shared routing contributes beyond simply increasing capacity or compute. To assess the additional value of TemporalFormer, we have now extended its comparison with mean pooling to BCIC2020-3, ISRUC, and CHB-MIT. TemporalFormer achieves higher means across all three reported metrics on each dataset, with balanced-accuracy gains of 1.34, 0.72, and 2.71 percentage points, respectively.

We believe that the consistent improvements in reported mean performance across six datasets support the effectiveness of TemporalFormer. This module is one of several contributions of TRACE, alongside its multi-horizon autoregressive pre-training framework and cross-channel shared routing design.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.