acceptodds
Under review as a conference paper at ICLR 2027

NeuroAct: Advancing EEG Analysis and Decoding via Dual-Loop Autonomous Research

Abstract

EEG analysis and decoding underpin brain-computer interfaces, cognitive assessment, and sleep monitoring. Fixed pipelines and reward-driven search do not explicitly determine how signal structure, representations, predictor behavior, and training dynamics contribute to performance or generalization gaps, leaving intervention choice underdetermined. We introduce NeuroAct, an autonomous research agent framework for EEG analysis and decoding. NeuroAct implements **dual-loop autonomous research** across EEG foundation models, task-specific predictors, and model combinations. In an inner loop, research, development, and review roles use evidence on signal structure, representations, predictor behavior, and training dynamics to test hypotheses, select analyses or interventions, and refine candidates using performance, stability, and cost feedback. An outer loop distills outcomes into conditional, testable recommendations with task and model conditions, then validates and revises them on subsequent tasks. We evaluate four settings: (A) EEG General Analysis, (B) Cross-task Model Optimization, (C) Single-task Open Research, and (D) Collaborative Prediction. In Setting A, NeuroAct's 80.88 macro-average exceeds BrainAgent's 63.13 by 17.75 points. In Setting B, NeuroAct raises EEGPT's mean balanced accuracy (BA) by 7.34 points and HBN Pearson's (r) from 0.6894 to 0.7362; in Setting C, it improves ADFTD BA by 16.33 points; and in Setting D, it raises Workload BA from 79.69% to 85.02%. Experience reuse raises BCIC-IV-2a BA from 34.66% to 50.81% while reducing LLM tokens by 64.02%, indicating recommendations improve cross-task performance. NeuroAct enables evidence-guided EEG research improved by cross-task experience.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG Analysis and Decoding, Autonomous Research Agents, Machine Learning Engineering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AFFJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on EEG analysis and decoding. Different EEG tasks may need different models and strategies, while traditional methods often use fixed pipelines or mainly rely on performance scores to guide the search. The authors propose NeuroAct, an agent framework with a dual-loop structure. Instead of just trying different methods and checking the final scores, it also uses intermediate steps like generating hypotheses, analyzing problems, and testing possible solutions. It also stores and updates experience from previous tasks to help with new ones. The experiments cover several EEG tasks and show good results.

Critical strengths
  1. The experiments cover different EEG tasks and settings, with generally good results. The appendix also includes some detailed cases.

  2. The idea of using intermediate analysis and hypothesis testing to guide research is interesting. I think it could also be useful for scientific problems beyond EEG.

Critical weaknesses
  1. The method is mostly a combination of existing agent frameworks, EEG tools, and memory.

  2. The diagnosis and hypothesis testing rely heavily on the LLM, and there is little evidence showing how reliable these intermediate judgments are, especially when they are wrong.

  3. There is still a gap between the claims and the experimental evidence. The results show that the system works well, but do not clearly show that the gains come from the proposed dual-loop reasoning rather than the tools, domain knowledge, or search space.

Questions for authors
  1. I am still not sure what the core methodological novelty is. The framework seems to rely a lot on the LLM's reasoning ability, domain knowledge, and tools. Since hypothesis generation and testing are central to this kind of research, I would like to understand what NeuroAct adds beyond these existing capabilities.

  2. Following the first point, final performance is easy to evaluate with clear metrics, but how are intermediate hypotheses actually tested? What evidence is used to support or rule out a hypothesis? Is there any explicit updating rule, such as Bayesian updating, or is this mainly decided by the LLM? How is the reliability of these judgments checked?

  3. Following the second point, how do the validation results affect future hypothesis generation? Is the feedback mainly kept in the prompt or research history, or is there a more explicit way to update the hypotheses being considered? How does the system decide which hypotheses to keep, reject, or explore next?

  4. I did not see a comparison with a simpler feedback-driven agent using the same tools, domain knowledge, search space, and budget. How much of the improvement comes from the proposed dual-loop mechanism itself?

  5. From the results, the evidence for experience reuse seems stronger than that for experience revision. How can we tell that revision itself helps? Are there repeated experiments showing consistent gains from revised experience?

Reviewer_pvhpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NeuroAct, an autonomous research-agent framework for EEG analysis and decoding. Its inner loop coordinates Researcher, Developer, and Reviewer roles to diagnose model behavior, implement interventions, and evaluate results. Its outer loop distills outcomes into conditional recommendations and reuses or revises them on subsequent tasks. The evaluation covers general EEG analysis, cross-task model optimization, single-task open research, and collaborative prediction across multiple datasets, model families, and LLM backends. The paper reports improvements in predictive performance and reductions in research cost through experience reuse.

Critical strengths
  1. The framework addresses a relevant problem: autonomous research agents should use diagnostic evidence and revise transferable recommendations rather than perform unconstrained trial-and-error search.
  2. The paper makes a substantial experimental effort across multiple EEG tasks, models, agent backends, capability ablations, and research-cost measurements.
Critical weaknesses
  1. The evaluation combines heterogeneous tasks, metrics, tool permissions, model pools, and baseline adaptations. Some Setting A comparisons use source-reported results, while other baselines are adapted to the authors’ environment. These differences make it difficult to attribute the reported gains specifically to the dual-loop design.
  2. The autonomous research process has substantial degrees of freedom, but the paper provides limited evidence about run-to-run variance. A single agent trajectory can benefit from fortunate model choices, code changes, or experimental decisions, so deterministic predictive metrics do not capture the uncertainty of the research procedure.
Questions for authors

1.Please repeat the principal mechanism comparison over multiple independent agent runs with different LLM seeds or sampling states and report the variance in final performance, research cost, and selected interventions. 2.Can the authors provide a matched-budget and matched-tools comparison in which NeuroAct, CodeAct, and the strongest competing agent receive the same models, data interfaces, token budget, GPU budget, and code-editing permissions? This would materially affect the attribution of gains. 3.Please evaluate the outer loop against no-experience, randomly retrieved experience, and unverified-summary controls. The current results should establish that conditional recommendation revision contributes beyond simply providing additional textual context. 4.Can the authors test experience transfer on a task or dataset excluded from experience construction and framework development? Otherwise, the reported transfer may reflect adaptation within the same benchmark collection. 5.Please describe the technical mechanism that prevents access to test labels, test metrics, or test-derived decisions during autonomous research. Are the complete tool logs auditable, and were any failed or discarded agent runs omitted from the reported results?

Reviewer_AbT6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents NeuroAct, an LLM multi-agent framework for EEG analysis and decoding. In an inner loop, a Coordinator directs Researcher, Developer, and Reviewer roles that use EEG analysis tools, runtime skills, and a code workspace to test and refine models. In an outer loop, outcomes are distilled into conditional recommendations that are reused and revised on later tasks. The evaluation has four settings: BrainBench analysis, joint training of one EEG-FM across five tasks, open single-task model development, and combining predictions from a model pool. It uses six datasets and three LLM backends, and adapts CodeAct, BrainAgent, and BrainPilot as baselines.

Critical strengths

The evaluation is broad: four settings, six datasets, several EEG-FMs, and three LLMs. It reports research cost (tokens, GPU hours, time) and includes a ten-run repeatability study, which few agent papers do. It also discloses known pretraining overlap for PhysioMI and HBN.

Experience is stored as structured, conditional recommendations with explicit applicability and reassessment conditions, which makes agent decisions more auditable than free-text memory.

Critical weaknesses

The claim that EEG-specific diagnostics drive the gains is asserted rather than shown. Most adopted interventions are generic ML fixes: class weighting, threshold shifts, a flattened head, a lower learning rate, and noise augmentation. Several headline gains change BA but not AUROC, which suggests decision-rule changes rather than better models. Many initial Setting B models sit at majority-class chance, so part of the improvement may be repairing weak starting points.

The experience loop is supported by four single-run comparisons, all on motor imagery, and two show little or no effect. The ablations use only CBraMod on three datasets. Clinical tasks are evaluated per window rather than per subject.

I would like to see comparisons with NeuroWeaver and EEG Agent, which are not contemporaneous. Comparisons with a general ML engineering agent (e.g., AIDE or MLE-STAR) and with a simple hyperparameter-search baseline under the same budget would show what the EEG-specific design adds.

The w/o experience BCIC-IV-2a run (34.66%, Tab. 25) is far below the paper’s own repeatability mean for the same setup (56.34 ± 1.76%, App. H), so the transfer gain may reflect one failed run. The Full ablation row matches the App. H means even though the caption says best. The ablations are single runs, some within one SD of Full. Several results are described as the best or highest evaluated candidate or study, and it is unclear whether these were selected on test data. Some baselines are at or below chance, including AUROC below 0.5 on ADFTD (Tab. 13), which points to interface problems. Baselines also had different interfaces and possibly different budgets.

Questions for authors

Why is the “w/o experience” BCIC-IV-2a result so far below the App. H mean? Did the setups differ?

Were the “best” or “highest” results selected on validation or test data, and how many studies were run per setting?

Did baselines get the same budget, member pools, and interfaces? Why is baseline AUROC below 0.5 on ADFTD?

For Setting A, what reasoning settings and token use did NeuroAct have, and were the tools or skills developed using BrainBench?

Could you:

  1. run multiple independent runs for the experience and ablation studies.
  2. add NeuroWeaver, a general ML agent, and a hyperparameter-search baseline with validation-tuned thresholds.
  3. report subject-level metrics and confidence intervals.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.