acceptodds
Under review as a conference paper at ICLR 2027

AutoBCI: Forecast-Guided Agentic Neural Architecture Discovery for EEG-Based Brain–Computer Interfaces

Abstract

EEG-based brain–computer interfaces support a broad range of applications, yet designing decoding architectures that perform well across diverse tasks remains challenging. We introduce AutoBCI, an agentic framework in which a Designer Agent and a Forecaster Agent support the discovery and selection of EEG decoding architectures across tasks. The Designer Agent performs Pool-Guided Architecture Discovery (PGAD), generating and refining architectures through training and validation across multiple EEG tasks, such as emotion recognition, motor imagery, and sleep staging. The Forecaster Agent performs Performance Estimation from Early Knowledge (PEEK), using architecture code, the training protocol, and early learning curves to predict full-budget validation performance and select promising candidates for continued training. Across 14 EEG datasets spanning motor imagery, emotion recognition, and sleep staging, we evaluate AutoBCI with six LLMs, including Opus 5.5 and GPT 5.6 Sol, and compare the architectures selected by the search procedure against ten baselines: six conventional EEG models and four foundation models. The architecture discovered by AutoBCI with Claude Opus 5.5 achieves 64.16% average test balanced accuracy (bAcc), compared with 63.87% for REVE, the strongest baseline on this metric. Using ten observed epochs, PEEK reduces mean absolute error in predicting average validation bAcc from 2.20 to 1.36 percentage points, a 38.1% reduction relative to the best-observed-score baseline.

Primary area:
applications to neuroscience & cognitive science
Keywords:
electroencephalography (EEG), brain–computer interfaces, neural architecture search, agentic discovery, performance forecasting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jChwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AutoBCI, an LLM-guided framework for automated EEG architecture discovery across motor imagery, emotion recognition, and sleep staging. It combines Pool-Guided Architecture Discovery (PGAD) for iterative architecture generation and refinement with Performance Estimation from Early Knowledge (PEEK) for early performance prediction and candidate selection. Experiments across 14 EEG datasets and six LLMs demonstrate competitive performance of the discovered lightweight architectures and explore the potential for reducing search costs.

Critical strengths

S1. Meaningful Research Problem and Unified Cross-Task Architecture Search

Automatically designing lightweight architectures for heterogeneous EEG tasks while reducing search costs is a meaningful research direction. The paper establishes a unified LLM-guided architecture search and selection framework across three distinct EEG task families.

S2. Comprehensive Evaluation and Detailed Reporting

The experiments cover diverse EEG datasets and multiple LLMs, with a generally reasonable evaluation protocol. Data preprocessing, training configurations, and architecture implementations are documented in detail. Iterative performance improvements are also observed across multiple LLM-guided searches.

S3. Competitive Lightweight Architectures

AutoBCI discovers architectures with fewer than 1M parameters, trained from scratch. The best architecture achieves 64.16% average balanced accuracy, slightly exceeding the pretrained REVE model with approximately 69M parameters. This demonstrates the practical potential of automated lightweight EEG architecture discovery.

Critical weaknesses

W1. Architectural Convergence and Potential Search Bias

Architectures discovered by different LLMs appear to favor familiar designs, including Inception-style modules, multiscale CNNs, and FBCNet-style statistical pooling. It would be important to investigate whether this similarity reflects prompt-induced bias or the LLMs' preferences for existing architectures.

W2. Evaluation of PEEK's Architecture Selection Capability

The paper emphasizes MAE reduction to demonstrate PEEK's predictive effectiveness. However, lower prediction error does not necessarily imply better architecture ranking. Although Top-k Winner Retention is reported, ranking-based metrics such as Kendall's tau could provide further insight into PEEK's candidate selection capability.

W3. Limited Discussion of Related Work

Papers like RZ-NAS and SAGE-NAS have explored LLM-guided architecture search with iterative feedback and low-cost performance evaluation. In particular, SAGE-NAS integrates LLM-driven architecture generation and fast performance prediction within a closed-loop search framework. A more detailed comparison would help clarify the distinctive contributions of PGAD and PEEK.

W4. Cross-Task Optimization Objective and Performance Balance

The three EEG task families differ in dataset composition and classification characteristics. Optimizing their equally weighted average balanced accuracy may obscure uneven improvements across tasks. Further discussion of how this objective reflects cross-task adaptability would be valuable.

W5. Limited Comparison with Alternative NAS Strategies

The paper lacks direct comparisons with alternative NAS search strategies, such as fresh architecture generation by LLM agents without iterative refinement or Bayesian optimization. Such comparisons would help establish the benefits of PGAD's search mechanism over existing alternatives.

Questions for authors

Q1 (W1). Have the authors examined whether the architectural similarities across LLMs arise from prompt design or learned architectural preferences?

Q2 (W2). Could the authors report Kendall's tau or similar ranking metrics to further evaluate PEEK's architecture selection capability?

Q3 (W3, W5). What distinguishes PGAD and PEEK from RZ-NAS and SAGE-NAS? Could the authors provide comparisons with simpler NAS strategies under comparable search budgets?

Q4 (W4). How does the equal-task weighting affect architecture selection? Are performance improvements reasonably balanced across the three task families or specific datasets?

4 comments

Q1 / W1. Prompt-induced bias versus learned architectural preferences.

Thank you for asking whether the shared architectural motifs are induced by the prompts. The prompt is neutral with respect to the architectural families discussed by the reviewer: it does not name or recommend Inception, multiscale CNNs, FBCNet, or statistical pooling, and supplies no starting architecture. It specifies the decoding tasks, input/output interface, parameter budget, and implementation constraints, including the exclusion of recurrent layers. The first round generates eight independent proposals. Moreover, the resulting champions use different aggregation mechanisms, including covariance, attention, and mean–standard-deviation pooling. Thus, the shared motifs are not explicitly prescribed by the prompt; attributing their emergence specifically to learned architectural preferences would require a separate controlled analysis.

Q2 / W2. Ranking metrics and PEEK's selection capability.

Thank you for suggesting ranking metrics to assess candidate selection beyond prediction error.

To assess selection directly, we report ranking correlation and selection regret alongside forecasting error. At ten observed epochs, the analysis covers 36 paired rounds and 269 candidates; metrics use the combined validation score and are averaged equally across rounds.

MethodMAE (pp)Kendall's tauTop-2 retention (%)Top-2 regret (pp)
Best observed score2.2030.73086.10.036
PEEK1.3640.73383.30.041

Regret measures the full-budget best balanced accuracy (bAcc) minus the best among retained candidates. PEEK reduces prediction error by 38.1%, while Kendall's tau is similar and top-2 retention is slightly lower. Thus, the observed improvement is in full-budget score prediction; the ranking and retention results do not establish better top-2 selection than the early-score baseline.

To connect round-level selection to the final model, we also replayed screening on these fixed candidate pools, selected final architectures by validation alone, and evaluated their saved seed-0 checkpoints on all three test tasks. PEEK preserves five of six final champions, including Opus, with unchanged test bAcc. The Gemini champion changes, reducing its test bAcc by 0.47 pp; the mean change across the six backends is −0.079 pp. Early-score screening selects the same final champions.

This policy would avoid 591 full-budget task runs and 54.9% of task-epochs. Scaling recorded training durations and adding 26.3 minutes of measured forecasting latency gives 32.30 versus 68.58 serial-equivalent hours, a 52.9% estimated reduction. Training costs are estimated by linear scaling within the fixed candidate pools, with forecasting added without overlap; they are not measured end-to-end GPU savings. Separately, three two-round adaptive pilots confirm that PEEK-selected first-round parents supply the evidence for subsequent generation; their six forecast calls took 54.9 seconds in total. Final test-performance changes are measured in the completed fixed-pool replay, rather than these partial adaptive runs.

Q3(a) / W3. Distinction from RZ-NAS and SAGE-NAS.

Thank you for pointing us to these closely related approaches and requesting a clearer comparison.

RZ-NAS combines LLM reflection with training-free evaluation. SAGE-NAS combines an LLM agent with a graph evaluator using topology and zero-cost statistics. Both establish precedents for LLM-guided search with inexpensive evaluation.

AutoBCI couples architecture refinement to measured performance across heterogeneous EEG tasks. PGAD ranks a historical parent pool by combined task validation scores and supplies parent code and task-specific learning curves for refinement. PEEK uses an LLM to predict full-budget performance from code, training specifications, and measured early curves. Its evaluation signal therefore includes observed training dynamics, whereas RZ-NAS uses training-free metrics and SAGE-NAS fuses graph topology with zero-cost statistics. The resulting architecture code is shared across MI, emotion, and sleep and trained independently for each task family.

Q3(b) / W5. Simpler NAS strategies under comparable budgets.

Thank you for suggesting independent generation and alternative search strategies to isolate the contribution of PGAD's refinement mechanism.

Independent fresh proposals from the same LLM would directly test the contribution of feedback-guided refinement: both settings can generate architecture code from a task description without a supplied starting network. PGAD combines these capabilities, beginning with eight independent proposals and then combining parent-based refinement with fresh explorations. A comparison that removes refinement while retaining the same LLM, proposal budget, data splits, training protocol, and validation objective would isolate its incremental benefit. Bayesian optimization would additionally require a specified architecture representation and proposal domain.

Our completed evaluations establish the effectiveness of the combined procedure: the frozen Opus architecture uses fewer than 0.5M parameters and exceeds REVE by 0.75 pp in the three-seed comparison. The additional adaptive pilots verify that selected parent evidence feeds into subsequent proposals. Neither result isolates the causal contribution of refinement. The independent-generation and Bayesian-optimization comparisons have not been performed; we therefore frame the contribution around the demonstrated quality of the discovered architectures, without claiming that PGAD outperforms those search alternatives.

Q4 / W4. Equal-task weighting and performance balance.

Thank you for asking how the aggregation objective affects task balance and architecture selection.

Equal-task weighting gives MI, emotion, and sleep equal influence despite their different dataset counts. We average dataset-level bAcc within each task, then weight the three tasks equally. Weighting all 14 datasets equally would instead give MI 8/14 of the objective and emotion and sleep 3/14 each.

Retraining the frozen architecture on seeds 0, 1, and 2 gives 64.03 ± 0.23% average test bAcc. Against the ten manuscript baselines, AutoBCI has the highest observed mean bAcc on MI and sleep, while REVE leads on emotion:

ModelMI bAcc (%), mean ± SDEmotion bAcc (%), mean ± SDSleep bAcc (%), mean ± SD
EEGNet56.87 ± 0.2836.54 ± 1.2866.64 ± 1.24
DeepConvNet60.75 ± 0.8539.65 ± 2.4672.13 ± 0.59
TSception54.70 ± 1.0043.13 ± 0.3866.44 ± 0.51
FAST59.73 ± 0.6135.19 ± 0.3671.87 ± 0.19
EEG-Conformer64.90 ± 0.5446.10 ± 1.0172.22 ± 0.19
AttnSleep62.45 ± 1.0139.07 ± 1.5273.74 ± 0.45
LaBraM46.66 ± 1.1249.20 ± 0.5671.10 ± 0.27
CBraMod47.50 ± 1.1639.63 ± 1.2869.20 ± 0.37
LEAF67.16 ± 0.1048.81 ± 0.2871.92 ± 0.58
REVE67.03 ± 0.7951.70 ± 0.4771.10 ± 0.07
AutoBCI (Opus)68.42 ± 0.2549.68 ± 0.1373.98 ± 0.33

Entries report mean ± sample SD across seeds 0, 1, and 2, using 40 epochs per task and model-specific training settings. Conventional baselines in the original manuscript used 100 epochs.

Relative to REVE, task-mean bAcc changes are +1.39 pp on MI, −2.01 pp on emotion, and +2.88 pp on sleep. Improvements occur on 6/8 MI, 1/3 emotion, and 3/3 sleep datasets, with the largest decrease on SEED-3 (−6.80 pp). The gains span MI and sleep, with an emotion trade-off concentrated particularly on SEED-3.

To test selection sensitivity, we reweighted the saved full-budget validation scores of all 269 complete candidates, keeping each search's candidate pool and task checkpoints fixed. The Opus gap measures how far its original champion falls below the best candidate in its pool under each weighted validation objective.

WeightingMI / Emotion / SleepChampions unchangedOriginal Opus rankOpus gap to best (pp)
Equal task1/3, 1/3, 1/36/610.000
Equal dataset8/14, 3/14, 3/144/620.156
MI emphasis0.50, 0.25, 0.254/620.054
Emotion emphasis0.25, 0.50, 0.256/610.000
Sleep emphasis0.25, 0.25, 0.506/610.000

All six champions remain selected when emotion or sleep receives half the weight. Under MI emphasis and equal-dataset weighting, four remain unchanged; the original Opus and GPT champions rank second in their pools, with Opus gaps of 0.054 and 0.156 pp, respectively. This validation-only replay measures selection sensitivity within the original candidate pools; it does not rerun adaptive search under alternative weights.

Thank you again for your valuable feedback. We hope our responses have addressed your concerns. If any issues remain, we would appreciate further clarification and the opportunity to address them.

Reviewer_uRhAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AutoBCI, an LLM-driven framework for discovering a shared EEG decoding architecture across emotion recognition, motor imagery, and sleep staging. A Designer Agent performs pool-guided architecture generation and refinement, while a Forecaster Agent predicts full-budget validation performance from architecture code, training specifications, and early learning curves. The study evaluates six LLMs, 14 EEG datasets, and ten conventional or foundation-model baselines. The selected architecture slightly exceeds the strongest baseline in average balanced accuracy, while the forecaster reduces prediction error relative to an early-score baseline.

Critical strengths
  1. The experimental scope is substantial: the search covers multiple EEG task families, datasets, LLM backends, conventional models, and foundation models under a common architecture-evaluation protocol.
  2. Separating architecture proposal from early-performance forecasting is sensible, and the paper evaluates both forecasting error and whether promising candidates survive early screening.
Critical weaknesses
  1. The final architecture improves average balanced accuracy from 63.87% to 64.16%, a margin of only 0.29 percentage points. Given the large search space and repeated architecture selection, this result does not convincingly establish that the discovered architecture is meaningfully better than the strongest baseline.
  2. All 14 datasets participate in the architecture-discovery objective. Consequently, the paper does not demonstrate that AutoBCI discovers transferable architectural principles rather than fitting its search decisions to a fixed collection of benchmarks.
Questions for authors

1.Please repeat the comparison between the selected AutoBCI architecture and the strongest baselines over multiple training seeds and provide paired uncertainty or significance estimates. Is the reported 0.29-point improvement robust to training randomness? 2.Can the authors perform a leave-one-dataset-out or leave-one-task-family-out search evaluation, in which the architecture is selected without access to the held-out dataset and is then evaluated on it? This is the most direct test of the claimed cross-task architecture generalization. 3.Please compare PGAD against a compute-matched random search, evolutionary search, or non-LLM architecture search using the same candidate budget and training protocol. Without this comparison, the benefit of using an LLM as the proposal operator remains unclear. 4.Beyond forecasting MAE, please report candidate-selection regret and total GPU savings: how often does PEEK retain the true best candidate, what performance is lost by early stopping, and how much full training does it avoid? 5.The search evaluates many candidates and six LLMs before selecting the best architecture. Please clarify the effective number of selection attempts and whether a nested validation procedure or an untouched meta-test set was used to control benchmark-level selection bias.

5 comments

Q1 / first weakness. Multi-seed robustness and statistical significance.

Thank you for requesting repeated-seed comparisons and paired uncertainty estimates.

The completed three-seed comparison strengthens the evidence: AutoBCI exceeds REVE by 0.75 percentage points in balanced accuracy (bAcc), with gains in every seed. We retrained the frozen Opus architecture from scratch on seeds 0, 1, and 2, preserving its code, original data splits, and 40-epoch protocol. Against the ten manuscript baselines, AutoBCI ranks first on both mean bAcc and Cohen's Kappa:

ModelbAcc (%), mean ± SDCohen's Kappa, mean ± SD
EEGNet53.35 ± 0.830.318 ± 0.015
DeepConvNet57.51 ± 1.150.384 ± 0.015
TSception54.76 ± 0.410.356 ± 0.007
FAST55.60 ± 0.280.362 ± 0.005
EEG-Conformer61.07 ± 0.150.448 ± 0.003
AttnSleep58.42 ± 0.350.397 ± 0.004
LaBraM55.65 ± 0.490.346 ± 0.007
CBraMod52.11 ± 0.310.298 ± 0.007
LEAF62.63 ± 0.280.472 ± 0.002
REVE63.28 ± 0.190.481 ± 0.005
AutoBCI (Opus)64.03 ± 0.230.488 ± 0.005

Within each seed, dataset scores are averaged within each task, then the three tasks are weighted equally. Entries report mean ± sample SD across three seeds. All models use a 40-epoch budget per task, retaining their respective training settings. This is a new repeated-run comparison; the original manuscript used 100 epochs for conventional baselines.

The original search used seed 0. With the architecture frozen, seeds 1 and 2 exceed REVE by 0.70 and 0.73 pp, respectively; the lowest AutoBCI score across all three reruns (63.76%) exceeds the highest REVE score (63.43%). Using the three seed-matched aggregate scores as the statistical units, two-sided paired t-tests give:

MetricAutoBCI − REVE (95% CI)pHolm-adjusted p
bAcc (pp)+0.750 [0.593, 0.908]0.002380.00714
Cohen's Kappa+0.00709 [0.00314, 0.01103]0.016350.03269
Weighted F1 (pp)+0.667 [0.279, 1.054]0.017790.03269

All three gains remain significant after Holm correction across the three metrics, conditional on the paired t-test assumptions with three seed pairs. Weighted F1 is 65.43 ± 0.33% for AutoBCI versus 64.76 ± 0.37% for REVE. The practical gain combines this consistent performance with fewer than 0.5M parameters, trained from scratch, versus 69.19M for REVE—over two orders of magnitude fewer parameters.

Q2 / second weakness. Generalization to datasets or task families excluded from search.

Thank you for suggesting evaluation on datasets and task families excluded from architecture discovery.

We added external evaluations on ADHD and Dreyer2023, excluded from architecture discovery and covering a new task and a new MI dataset. ADHD_AliMotie introduces a classification task outside the original MI, emotion, and sleep search objectives. MI_Dreyer2023 tests transfer to a new dataset within the MI task family. Both targets were excluded from the entire architecture search and supplied no training or validation feedback during discovery.

To specify the external evaluation inputs, we use the following datasets and preprocessing:

For ADHD, we apply percentile clipping (0.1–99.9%), median/IQR scaling, and mapping to a common 65-channel montage with interpolation of missing electrodes. Our evaluation loader additionally standardizes each trial/channel over time for both datasets and all models.

All six original validation-selected champions were frozen before these evaluations, with no architecture modification or further search. Each architecture was trained from scratch on each target dataset using seeds 0, 1, and 2 and the fixed 40-epoch protocol. REVE and LEAF used the same target splits and epoch budget, retaining their pretrained initialization and model-specific training settings. Checkpoints were selected on target validation data before test evaluation. Results below are test bAcc (%), mean ± sample SD; ranks use the unrounded means.

ModelADHD_AliMotieRankMI_Dreyer2023Rank
AutoBCI (Opus)75.82 ± 1.80377.15 ± 0.251
AutoBCI (Sonnet)72.66 ± 4.29571.64 ± 0.106
AutoBCI (DeepSeek)75.33 ± 2.65469.07 ± 1.098
AutoBCI (Gemini)76.87 ± 3.18174.68 ± 0.735
AutoBCI (GPT)65.85 ± 6.13876.16 ± 1.402
AutoBCI (Qwen)76.14 ± 1.97269.81 ± 0.967
REVE71.94 ± 1.18676.13 ± 1.433
LEAF71.39 ± 0.52775.86 ± 0.524

The original Opus champion exceeds both foundation-model baselines on both datasets: its mean bAcc gains over REVE are +3.87 pp on ADHD and +1.02 pp on Dreyer2023, and over LEAF +4.43 pp and +1.28 pp, respectively. Opus was selected using validation scores on the original benchmarks, before any of these external evaluations. These results provide direct evidence that the discovered architecture transfers beyond the benchmarks that guided its discovery.

This external-transfer protocol tests the central separation requested by the reviewer: architecture selection precedes access to the target dataset or task family. Both targets were held out from architecture discovery.

Q3. Compute-matched alternative search strategies.

Thank you for suggesting a controlled comparison with alternative search strategies.

PGAD constructs executable architectures directly from a decoding problem and its constraints, without a user-supplied starting network or a predefined catalogue of candidate topologies. Its first round generates eight independent architectures from the task description, input/output interface, and resource and implementation constraints. Subsequent rounds combine refinement using measured parent evidence with fresh explorations.

Random search samples a specified architecture space, while evolutionary search develops an initial population through specified mutation or crossover operators. Both can change network structure; their results depend on the chosen representation, permitted operations, and sampling or initialization rules. A controlled comparison should therefore specify these choices alongside the candidate budget and training protocol. Such a comparison would help quantify the contribution of the LLM proposal operator.

We have not performed this comparison, so our evidence supports the effectiveness of the complete task-to-code workflow rather than superiority over alternative search operators. This workflow discovers a decoder with fewer than 0.5M parameters, consistent three-seed gains over REVE, and transfer to ADHD and Dreyer2023 without further architecture search.

Q4. Selection regret, winner retention, and GPU savings.

Thank you for asking us to connect forecasting accuracy to candidate retention, final performance, and computational savings.

We agree that forecasting accuracy must be connected to the quality retained for the compute spent. We therefore added a controlled screening analysis and test evaluation: given the same proposed architectures, how much training could be avoided, and would screening change the final selected model? Using fixed candidate pools makes every candidate's full-budget outcome available, so both discarded winners and selection regret can be measured directly.

Across 36 rounds and 269 candidates, we retrospectively replay top-two selection using only the first 10 epochs and score the retained candidates using their saved 40-epoch outcomes. Regret is the gap between the best full-budget validation bAcc and the best retained candidate, averaged across rounds.

Screening policyRound winner retained (%)Mean regret (pp)Full-budget task runsEstimated training job-hours
Full training100.00.00080768.58
Early score, top 286.10.03621631.86
PEEK, top 283.30.04121631.86

Missing a round winner does not necessarily change the final architecture. After freezing promotions and selecting each search's final architecture by validation alone, we evaluated the original seed-0 checkpoints on all three test tasks. PEEK preserves five of six final champions, including Opus, with unchanged test bAcc. Gemini loses 0.47 pp; the mean change across the six backends is −0.079 pp. Early-score screening selects the same final champions and has slightly better round-level retention and regret.

The corresponding budget would avoid 591 full-budget task runs and 54.9% of task-epochs. Adding 26.3 minutes of measured forecasting latency gives 32.30 versus 68.58 serial-equivalent hours, a 52.9% estimated reduction. Training time is scaled from recorded full-run durations, with forecasting added without overlap.

To complement this fixed-pool analysis, we also exercised PEEK within an adaptive proposal loop across three runs and two rounds per run. These additional runs used a 16-epoch screening window. Six forecast calls took 54.9 seconds in total, and the two selected first-round candidates supplied the parent evidence for second-round architecture generation. Of 48 proposed architectures, 47 completed early training on all three tasks; one timed out and was excluded from forecasting. This provides direct evidence that PEEK can guide subsequent proposals with modest observed forecasting latency.

The measured early-training times also support a separate cost estimate for these online pilots. Extrapolating each eligible candidate's 16-epoch duration to 40 epochs, while continuing only the selected candidates for 24 additional epochs, gives 10.96 versus 18.97 serial-equivalent hours—a 42.2% estimated reduction across the six rounds. This includes the measured forecast latency and charges the observed failed-candidate cost to both policies. The estimate assumes linear training-time scaling and successful continuation, holds the observed candidate pools fixed, and excludes proposal-generation costs.

Final test-performance changes are measured in the fixed-pool replay; the two-round adaptive pilots measure parent selection and forecasting latency.

Reviewer_X1wbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AutoBCI combines LLM-driven architecture generation with early performance forecasting to discover shared EEG decoders across multiple public datasets.

Critical strengths

The experimental scope is substantial, covering fourteen datasets, three task families, and six LLM-guided searches. The paper provides prompts, architecture lineages, failure logs, learning curves, and generated code, supporting reproducibility. Test sets are excluded from architecture search, while equal task weighting establishes a clear and consistent selection objective across heterogeneous decoding tasks.

Critical weaknesses

The best discovered architecture outperforms REVE by only 0.29 percentage points in balanced accuracy, without confidence intervals or statistical significance testing. Meanwhile, REVE performs better on weighted F1 and Cohen’s kappa, making the claimed superiority questionable. Furthermore, selecting the best architecture among 269 trained candidates introduces a substantial risk of winner’s curse, particularly without independent retraining across multiple random seeds. Each LLM also evaluates 48 architectures against the same validation pool, potentially leading to progressive validation overfitting despite keeping test sets untouched. The paper does not adequately assess this selection bias through nested validation or independent external datasets. Consequently, the reported improvement may reflect search variability rather than a meaningful architectural advantage.

Questions for authors
  • Is the 0.29-percentage-point improvement in balanced accuracy over REVE statistically significant?
  • How do the authors justify AutoBCI's superiority when REVE achieves better weighted F1 and Cohen’s kappa?
  • How do the authors account for winner’s curse when selecting the best architecture from 269 candidates, and mitigate potential validation overfitting during repeated architecture searches?
3 comments

Q1. Statistical significance of the 0.29-percentage-point improvement.

Thank you for asking us to quantify the robustness and uncertainty of the reported improvement.

We retrained the frozen Opus architecture from scratch on seeds 0, 1, and 2, keeping its code, original data splits, and 40-epoch protocol fixed. We compare its test balanced accuracy (bAcc) and Cohen's Kappa against the ten manuscript baselines with completed three-seed records.

ModelbAcc (%), mean ± SDCohen's Kappa, mean ± SD
EEGNet53.35 ± 0.830.318 ± 0.015
DeepConvNet57.51 ± 1.150.384 ± 0.015
TSception54.76 ± 0.410.356 ± 0.007
FAST55.60 ± 0.280.362 ± 0.005
EEG-Conformer61.07 ± 0.150.448 ± 0.003
AttnSleep58.42 ± 0.350.397 ± 0.004
LaBraM55.65 ± 0.490.346 ± 0.007
CBraMod52.11 ± 0.310.298 ± 0.007
LEAF62.63 ± 0.280.472 ± 0.002
REVE63.28 ± 0.190.481 ± 0.005
AutoBCI (Opus)64.03 ± 0.230.488 ± 0.005

Within each seed, dataset scores are averaged within each task, then the three tasks are weighted equally. Entries report mean ± sample SD across three seeds. This new comparison uses 40 epochs per task for all models, retaining their respective training settings; the original manuscript used 100 epochs for conventional baselines.

Across three training seeds, AutoBCI ranks first among all eleven models on both mean bAcc and Cohen's Kappa, exceeding the strongest baseline, REVE, by 0.75 percentage points and 0.007, respectively. This performance is achieved by a decoder trained from scratch with fewer than 0.5M parameters, compared with REVE's 69.19M—over two orders of magnitude fewer parameters.

The advantage is consistent across the repeated runs: AutoBCI exceeds REVE on both metrics in every seed-matched comparison, with bAcc gains of 0.70–0.82 percentage points. The architecture was frozen before retraining, and all three seeds are included. These results directly address the concern that the reported performance depends on a single favorable training seed and support the practical value of discovering compact EEG decoders.

Using the three seed-matched aggregate scores as the statistical units, a two-sided paired t-test gives a bAcc difference of +0.750 pp (95% CI [0.593, 0.908], p = 0.00238) and a Cohen's Kappa difference of +0.00709 (95% CI [0.00314, 0.01103], p = 0.01635). Both remain significant at 0.05 after Holm correction across bAcc, Cohen's Kappa, and weighted F1 (adjusted p = 0.00714 and 0.03269, respectively).

Q2. Comparison with REVE across metrics and model sizes.

Thank you for highlighting the importance of comparing performance across metrics alongside model size.

The discovered architecture uses fewer than 0.5M parameters, compared with 69.19M for REVE, and is trained from scratch. Its highest mean bAcc and Cohen's Kappa across the completed baseline comparison demonstrate competitive decoding with over two orders of magnitude fewer parameters than REVE.

bAcc is the predefined selection objective and weights class recalls equally. In the original manuscript, REVE led on weighted F1 (65.78% versus 65.67%) and Cohen's Kappa (0.495 versus 0.492). In the completed three-seed records, AutoBCI also has higher mean weighted F1: 65.43 ± 0.33%, compared with 64.76 ± 0.37% for REVE. The gain is +0.667 pp (95% CI [0.279, 1.054]; two-sided paired t-test, p = 0.01779) and remains significant after Holm correction across all three metrics (adjusted p = 0.03269). Together with Q1, these repeated runs favor AutoBCI on all three metrics; significance is conditional on the paired t-test assumptions with three seed pairs.

Q3. Winner's curse and repeated validation use.

Thank you for raising the concern about selection bias from repeated architecture evaluation.

The advantage survives training seeds that were never used during architecture search. The original search used seed 0. With the architecture and training protocol frozen, seeds 1 and 2 exceed REVE by 0.70 and 0.73 pp bAcc, respectively, in the new comparison. Across all three reruns, the lowest AutoBCI result (63.76%) exceeds the highest REVE result (63.43%). All seeds are included, and no further architecture selection is performed. This directly addresses whether the selected architecture's advantage depends on the favorable training realization used during search.

To separate adaptive selection from performance estimation, each search selects its champion by validation bAcc before test evaluation. The six searches make 288 proposals, with 269 completing all tasks, and all six champions are reported. Test scores never guide proposals, parent choice, or checkpoint selection. The original search used fixed partitions without nested outer validation.

To assess transfer beyond the repeatedly used discovery benchmarks, we froze all six original champions before evaluating ADHD_AliMotie and MI_Dreyer2023, which supplied no training or validation feedback during architecture search. Each architecture was trained from scratch on each target using seeds 0, 1, and 2. REVE and LEAF used the same target splits and 40-epoch budget, retaining their pretrained initialization and model-specific training settings; checkpoints were selected on target validation data before test evaluation.

To specify the external evaluation inputs, we use the following datasets and preprocessing:

For ADHD, we apply percentile clipping (0.1–99.9%), median/IQR scaling, and mapping to a common 65-channel montage with interpolation of missing electrodes. Our evaluation loader additionally standardizes each trial/channel over time for both datasets and all models.

The original Opus champion exceeds both REVE and LEAF on ADHD and Dreyer2023. Test bAcc (%, mean ± sample SD) is:

DatasetAutoBCI (Opus)REVELEAF
ADHD_AliMotie75.82 ± 1.8071.94 ± 1.1871.39 ± 0.52
MI_Dreyer202377.15 ± 0.2576.13 ± 1.4375.86 ± 0.52

ADHD tests a task outside the original search objectives, while Dreyer2023 tests a new dataset within MI. These external evaluations provide evidence beyond the architecture-selection benchmarks, complementing the training-seed robustness in Q1. They assess transfer after selection, rather than directly estimating validation-selection optimism.

Thank you again for your valuable feedback. We hope our responses have addressed your concerns. As the rebuttal period draws to a close, we would sincerely appreciate it if you could reconsider your score in light of our clarifications. If any concerns remain, we would be grateful for the opportunity to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.