acceptodds
Under review as a conference paper at ICLR 2027

EEGAgentBench: Benchmarking LLM Agents on Short- and Long-Horizon EEG Analysis

Abstract

Electroencephalography (EEG) analysis is evolving from short-segment classification toward long-horizon interpretation that demands iterative evidence accumulation, multi-step reasoning, and coordinated use of specialized signal-processing tools. Although large language models (LLMs) have recently shown promise as autonomous agents for EEG analysis, existing EEG agentic evaluations remain fragmented, covering limited tasks over narrow temporal horizons with inconsistent protocols, and providing no comprehensive assessment of agents' reasoning, tool-use, and workflow construction capabilities. To address this gap, we propose EEGAgentBench, a unified benchmark for systematically evaluating LLM agents on short- and long-horizon EEG analysis. EEGAgentBench spans six representative EEG applications ranging from knowledge question answering to sleep staging. It encompasses signal durations from 2 seconds to nearly 23 hours, with prediction targets ranging from class labels to event intervals and epoch-level sequences. This design supports unified evaluation across knowledge reasoning, short-horizon interpretation, long-horizon event detection, and sequential understanding. The benchmark further provides 10 deterministic EEG analysis tools that expose only task-relevant signal measurements. Agents must therefore select tools autonomously, accumulate evidence iteratively, and construct multi-step workflows. For evaluation, we benchmark 29 frontier LLMs from 15 model families. Results demonstrate that EEGAgentBench effectively distinguishes agent capabilities beyond model scale and inference cost, while revealing substantial limitations of current LLM agents in long-horizon EEG analysis, particularly in sustained evidence accumulation and multi-step reasoning.

Primary area:
datasets and benchmarks
Keywords:
EEG analysis, LLM agents, long-horizon reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HrHsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EEGAgentBench evaluates LLM agents across six EEG tasks, comprising 1,072 instances with recording durations from two seconds to nearly 23 hours. It provides ten deterministic signal-processing tools and benchmarks 29 models, analyzing task performance, tool use, inference cost, and failures on long recordings.

Critical strengths
  1. The benchmark covers diverse prediction formats, including classification, seizure localization, and full-recording sleep staging. A shared environment with deterministic tools provides a useful framework for comparing EEG agents across tasks and temporal scales.

  2. The evaluation combines broad model coverage with informative metrics and trajectory analyses. Subject-averaged seizure F1, temporal Dice, and sleep-stage kappa reveal complementary failure modes that aggregate accuracy would miss.

Critical weaknesses
  1. The long-horizon analysis does not adequately separate reasoning limitations from evidence acquisition and recording difficulty. Duration comparisons use different recordings, while case studies show tool-output limits followed by coarser measurements. Matched segments from the same recordings and a fixed, chunked feature-extraction baseline are needed to substantiate the central interpretation.

  2. T3 confounds seizure status with recording modality: the healthy examples are scalp EEG, whereas the ictal examples are intracranial EEG. Performance could therefore partly reflect acquisition differences rather than epileptic-pattern interpretation. A comparison within the same recording modality is needed.

  3. Difficulty filtering uses five models that subsequently appear in the evaluation, making the challenge distribution dependent on the screening panel. This substantially changes some tasks, removing 111 of 149 T1 questions. Rankings should be checked on the unfiltered pool or an independently selected subset.

  4. Reproducibility and ranking stability remain insufficiently characterized. Models use different decoding configurations, but exact sampling and reasoning settings are not reported. Single-run evaluation without uncertainty estimates further limits comparisons, particularly for small tasks and extreme-duration groups.

Questions for authors
  1. Does the duration-related performance decline persist on matched portions of the same recordings and when agents receive systematically extracted features through a fixed chunked workflow?

  2. How sensitive are the overall rankings to difficulty filtering and to replacing T3 with a comparison within the same recording modality?

Reviewer_Y4naOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a curated benchmark of EEG data consisting of examples from 6 existing datasets. The authors argue that this helps reduce issues withe existing benchmarks that occur because of inconsistent protocols, and lack of ways to address agents tool-use. The paper evaluates many (29) LLMs from 15 families.

Critical strengths

The paper addresses a reasonable gap about the standardization of data across existing public datasets. As we move to evaluating agents there is a need to revisit the contents, structure and formulation of benchmark datasets. The labor in preparing such datasets is non-trivial and I would like to give credit to the authors for that work.

Furthermore, the authors as diligent to test many different models in their analysis. This is certainly one of the more comprehensive lists of LLM comparisons I have seen.

Critical weaknesses

My main critique of this paper is the low amount of technical innovation or novelty for the reader to learn from. Curating an EEG dataset of this nature will be of interest to the communities studying EEG but the paper has very little additional insight for the machine learning community to garner.

The results and conclusions of the paper are fairly expected and there is little exposition analysis or innovation in those experiments (i.e., they lack some level of creativity or insight that would increase the interest of impact of the paper.

The authors do not seem to level human input in the creation of the dataset much, relying on auto-evaluations and difficulty estimation based on "panels" of models. As a result the paper lacks substantial insight that I would expect from an ICLR paper.

Questions for authors

Please explain if you considered additional human-expert cleaning/filtering or extension of these dataset, with annotation or meta data.

Please add more error bars/variances to your results to capture the performance of the models beyond average accuracy.

Please rearticulate the key machine learning contributions of this paper that would make it a fit for ICLR vs a more EEG/BCI readership.

Reviewer_65wiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Motivated by the observation that existing agentic evaluations cover few tasks over short time spans and use inconsistent protocols, the paper proposes EEGAgentBench, the first unified benchmark that evaluates general-purpose LLM agents on both short- and long-horizon EEG analysis through standardized tool-based workflows. The benchmark spans six EEG tasks, from knowledge question answering to sleep staging, with signal durations from 2 seconds to nearly 23 hours and outputs ranging from class labels to event intervals and epoch-level sequences. Agents can access the signal only through ten deterministic, non-parametric signal-processing tools. They must therefore choose tools, gather evidence over multiple steps, and build their own workflows, so a single setting tests knowledge reasoning, adaptive tool use, and workflow construction. The authors evaluate 29 frontier LLMs from 15 model families on task performance, tool-use behavior, computational efficiency, and long-horizon failure modes. This establishes baselines and shows that the benchmark separates agent capability from model scale and inference cost. It also shows that current agents do poorly on long-horizon analysis, mainly because they fail to build up evidence over many steps and to keep multi-step reasoning going.

Critical strengths

The paper evaluates a diverse set of models across different scales and model families with cost analysis, providing a useful comparison for EEG task deployment.

To our knowledge, this is the first benchmark that assesses the agentic ability of LLMs on EEG tasks. Organizing the tasks into three levels is useful for understanding where model capability breaks down.

Critical weaknesses

It is unclear whether the performance of the evaluated LLMs is limited by the small set of tools that the EEG environment provides or by the capability of the models themselves. A baseline that queries each open-source and proprietary model directly through its API would show how each model performs natively and would separate these two factors.

The motivation for a tool-based benchmark is not well established. The authors should report how LLMs perform without tool access. Some tasks may be solvable without tools, for example through few-shot prompting, and this would weaken the case for the proposed environment.

Questions for authors

Could the authors provide a more detailed error analysis of where and why LLMs fail on EEG tasks? In particular, it would help to know whether failures come mainly from perception errors, such as calling tools on irrelevant segments, or from reasoning errors, such as correctly identifying a pattern but drawing the wrong inference from it. Without this analysis, the paper reads more as a leaderboard than as a benchmark that gives insight into model behavior.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.