- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AutoBCI, an LLM-guided framework for automated EEG architecture discovery across motor imagery, emotion recognition, and sleep staging. It combines Pool-Guided Architecture Discovery (PGAD) for iterative architecture generation and refinement with Performance Estimation from Early Knowledge (PEEK) for early performance prediction and candidate selection. Experiments across 14 EEG datasets and six LLMs demonstrate competitive performance of the discovered lightweight architectures and explore the potential for reducing search costs.
Critical strengths
S1. Meaningful Research Problem and Unified Cross-Task Architecture Search
Automatically designing lightweight architectures for heterogeneous EEG tasks while reducing search costs is a meaningful research direction. The paper establishes a unified LLM-guided architecture search and selection framework across three distinct EEG task families.
S2. Comprehensive Evaluation and Detailed Reporting
The experiments cover diverse EEG datasets and multiple LLMs, with a generally reasonable evaluation protocol. Data preprocessing, training configurations, and architecture implementations are documented in detail. Iterative performance improvements are also observed across multiple LLM-guided searches.
S3. Competitive Lightweight Architectures
AutoBCI discovers architectures with fewer than 1M parameters, trained from scratch. The best architecture achieves 64.16% average balanced accuracy, slightly exceeding the pretrained REVE model with approximately 69M parameters. This demonstrates the practical potential of automated lightweight EEG architecture discovery.
Critical weaknesses
W1. Architectural Convergence and Potential Search Bias
Architectures discovered by different LLMs appear to favor familiar designs, including Inception-style modules, multiscale CNNs, and FBCNet-style statistical pooling. It would be important to investigate whether this similarity reflects prompt-induced bias or the LLMs' preferences for existing architectures.
W2. Evaluation of PEEK's Architecture Selection Capability
The paper emphasizes MAE reduction to demonstrate PEEK's predictive effectiveness. However, lower prediction error does not necessarily imply better architecture ranking. Although Top-k Winner Retention is reported, ranking-based metrics such as Kendall's tau could provide further insight into PEEK's candidate selection capability.
W3. Limited Discussion of Related Work
Papers like RZ-NAS and SAGE-NAS have explored LLM-guided architecture search with iterative feedback and low-cost performance evaluation. In particular, SAGE-NAS integrates LLM-driven architecture generation and fast performance prediction within a closed-loop search framework. A more detailed comparison would help clarify the distinctive contributions of PGAD and PEEK.
W4. Cross-Task Optimization Objective and Performance Balance
The three EEG task families differ in dataset composition and classification characteristics. Optimizing their equally weighted average balanced accuracy may obscure uneven improvements across tasks. Further discussion of how this objective reflects cross-task adaptability would be valuable.
W5. Limited Comparison with Alternative NAS Strategies
The paper lacks direct comparisons with alternative NAS search strategies, such as fresh architecture generation by LLM agents without iterative refinement or Bayesian optimization. Such comparisons would help establish the benefits of PGAD's search mechanism over existing alternatives.
Questions for authors
Q1 (W1). Have the authors examined whether the architectural similarities across LLMs arise from prompt design or learned architectural preferences?
Q2 (W2). Could the authors report Kendall's tau or similar ranking metrics to further evaluate PEEK's architecture selection capability?
Q3 (W3, W5). What distinguishes PGAD and PEEK from RZ-NAS and SAGE-NAS? Could the authors provide comparisons with simpler NAS strategies under comparable search budgets?
Q4 (W4). How does the equal-task weighting affect architecture selection? Are performance improvements reasonably balanced across the three task families or specific datasets?
4 comments
Q3(a) / W3. Distinction from RZ-NAS and SAGE-NAS.
Thank you for pointing us to these closely related approaches and requesting a clearer comparison.
RZ-NAS combines LLM reflection with training-free evaluation. SAGE-NAS combines an LLM agent with a graph evaluator using topology and zero-cost statistics. Both establish precedents for LLM-guided search with inexpensive evaluation.
AutoBCI couples architecture refinement to measured performance across heterogeneous EEG tasks. PGAD ranks a historical parent pool by combined task validation scores and supplies parent code and task-specific learning curves for refinement. PEEK uses an LLM to predict full-budget performance from code, training specifications, and measured early curves. Its evaluation signal therefore includes observed training dynamics, whereas RZ-NAS uses training-free metrics and SAGE-NAS fuses graph topology with zero-cost statistics. The resulting architecture code is shared across MI, emotion, and sleep and trained independently for each task family.
Q3(b) / W5. Simpler NAS strategies under comparable budgets.
Thank you for suggesting independent generation and alternative search strategies to isolate the contribution of PGAD's refinement mechanism.
Independent fresh proposals from the same LLM would directly test the contribution of feedback-guided refinement: both settings can generate architecture code from a task description without a supplied starting network. PGAD combines these capabilities, beginning with eight independent proposals and then combining parent-based refinement with fresh explorations. A comparison that removes refinement while retaining the same LLM, proposal budget, data splits, training protocol, and validation objective would isolate its incremental benefit. Bayesian optimization would additionally require a specified architecture representation and proposal domain.
Our completed evaluations establish the effectiveness of the combined procedure: the frozen Opus architecture uses fewer than 0.5M parameters and exceeds REVE by 0.75 pp in the three-seed comparison. The additional adaptive pilots verify that selected parent evidence feeds into subsequent proposals. Neither result isolates the causal contribution of refinement. The independent-generation and Bayesian-optimization comparisons have not been performed; we therefore frame the contribution around the demonstrated quality of the discovered architectures, without claiming that PGAD outperforms those search alternatives.
Q4 / W4. Equal-task weighting and performance balance.
Thank you for asking how the aggregation objective affects task balance and architecture selection.
Equal-task weighting gives MI, emotion, and sleep equal influence despite their different dataset counts. We average dataset-level bAcc within each task, then weight the three tasks equally. Weighting all 14 datasets equally would instead give MI 8/14 of the objective and emotion and sleep 3/14 each.
Retraining the frozen architecture on seeds 0, 1, and 2 gives 64.03 ± 0.23% average test bAcc. Against the ten manuscript baselines, AutoBCI has the highest observed mean bAcc on MI and sleep, while REVE leads on emotion:
| Model | MI bAcc (%), mean ± SD | Emotion bAcc (%), mean ± SD | Sleep bAcc (%), mean ± SD |
|---|---|---|---|
| EEGNet | 56.87 ± 0.28 | 36.54 ± 1.28 | 66.64 ± 1.24 |
| DeepConvNet | 60.75 ± 0.85 | 39.65 ± 2.46 | 72.13 ± 0.59 |
| TSception | 54.70 ± 1.00 | 43.13 ± 0.38 | 66.44 ± 0.51 |
| FAST | 59.73 ± 0.61 | 35.19 ± 0.36 | 71.87 ± 0.19 |
| EEG-Conformer | 64.90 ± 0.54 | 46.10 ± 1.01 | 72.22 ± 0.19 |
| AttnSleep | 62.45 ± 1.01 | 39.07 ± 1.52 | 73.74 ± 0.45 |
| LaBraM | 46.66 ± 1.12 | 49.20 ± 0.56 | 71.10 ± 0.27 |
| CBraMod | 47.50 ± 1.16 | 39.63 ± 1.28 | 69.20 ± 0.37 |
| LEAF | 67.16 ± 0.10 | 48.81 ± 0.28 | 71.92 ± 0.58 |
| REVE | 67.03 ± 0.79 | 51.70 ± 0.47 | 71.10 ± 0.07 |
| AutoBCI (Opus) | 68.42 ± 0.25 | 49.68 ± 0.13 | 73.98 ± 0.33 |
Entries report mean ± sample SD across seeds 0, 1, and 2, using 40 epochs per task and model-specific training settings. Conventional baselines in the original manuscript used 100 epochs.
Relative to REVE, task-mean bAcc changes are +1.39 pp on MI, −2.01 pp on emotion, and +2.88 pp on sleep. Improvements occur on 6/8 MI, 1/3 emotion, and 3/3 sleep datasets, with the largest decrease on SEED-3 (−6.80 pp). The gains span MI and sleep, with an emotion trade-off concentrated particularly on SEED-3.
To test selection sensitivity, we reweighted the saved full-budget validation scores of all 269 complete candidates, keeping each search's candidate pool and task checkpoints fixed. The Opus gap measures how far its original champion falls below the best candidate in its pool under each weighted validation objective.
| Weighting | MI / Emotion / Sleep | Champions unchanged | Original Opus rank | Opus gap to best (pp) |
|---|---|---|---|---|
| Equal task | 1/3, 1/3, 1/3 | 6/6 | 1 | 0.000 |
| Equal dataset | 8/14, 3/14, 3/14 | 4/6 | 2 | 0.156 |
| MI emphasis | 0.50, 0.25, 0.25 | 4/6 | 2 | 0.054 |
| Emotion emphasis | 0.25, 0.50, 0.25 | 6/6 | 1 | 0.000 |
| Sleep emphasis | 0.25, 0.25, 0.50 | 6/6 | 1 | 0.000 |
All six champions remain selected when emotion or sleep receives half the weight. Under MI emphasis and equal-dataset weighting, four remain unchanged; the original Opus and GPT champions rank second in their pools, with Opus gaps of 0.054 and 0.156 pp, respectively. This validation-only replay measures selection sensitivity within the original candidate pools; it does not rerun adaptive search under alternative weights.
Thank you again for your valuable feedback. We hope our responses have addressed your concerns. If any issues remain, we would appreciate further clarification and the opportunity to address them.
Q1 / W1. Prompt-induced bias versus learned architectural preferences.
Thank you for asking whether the shared architectural motifs are induced by the prompts. The prompt is neutral with respect to the architectural families discussed by the reviewer: it does not name or recommend Inception, multiscale CNNs, FBCNet, or statistical pooling, and supplies no starting architecture. It specifies the decoding tasks, input/output interface, parameter budget, and implementation constraints, including the exclusion of recurrent layers. The first round generates eight independent proposals. Moreover, the resulting champions use different aggregation mechanisms, including covariance, attention, and mean–standard-deviation pooling. Thus, the shared motifs are not explicitly prescribed by the prompt; attributing their emergence specifically to learned architectural preferences would require a separate controlled analysis.
Q2 / W2. Ranking metrics and PEEK's selection capability.
Thank you for suggesting ranking metrics to assess candidate selection beyond prediction error.
To assess selection directly, we report ranking correlation and selection regret alongside forecasting error. At ten observed epochs, the analysis covers 36 paired rounds and 269 candidates; metrics use the combined validation score and are averaged equally across rounds.
Regret measures the full-budget best balanced accuracy (bAcc) minus the best among retained candidates. PEEK reduces prediction error by 38.1%, while Kendall's tau is similar and top-2 retention is slightly lower. Thus, the observed improvement is in full-budget score prediction; the ranking and retention results do not establish better top-2 selection than the early-score baseline.
To connect round-level selection to the final model, we also replayed screening on these fixed candidate pools, selected final architectures by validation alone, and evaluated their saved seed-0 checkpoints on all three test tasks. PEEK preserves five of six final champions, including Opus, with unchanged test bAcc. The Gemini champion changes, reducing its test bAcc by 0.47 pp; the mean change across the six backends is −0.079 pp. Early-score screening selects the same final champions.
This policy would avoid 591 full-budget task runs and 54.9% of task-epochs. Scaling recorded training durations and adding 26.3 minutes of measured forecasting latency gives 32.30 versus 68.58 serial-equivalent hours, a 52.9% estimated reduction. Training costs are estimated by linear scaling within the fixed candidate pools, with forecasting added without overlap; they are not measured end-to-end GPU savings. Separately, three two-round adaptive pilots confirm that PEEK-selected first-round parents supply the evidence for subsequent generation; their six forecast calls took 54.9 seconds in total. Final test-performance changes are measured in the completed fixed-pool replay, rather than these partial adaptive runs.