acceptodds
Under review as a conference paper at ICLR 2027

BrainBench: Benchmarking Large Language Models for Comprehensive EEG Understanding

Abstract

Large language models (LLMs) are increasingly being explored for electroencephalography (EEG) analysis, yet their ability to support comprehensive EEG understanding remains poorly understood. Existing evaluations largely focus on isolated decoding tasks or demonstrations tailored to specific systems, capturing only limited aspects of EEG competence. In practice, EEG analysis requires a system to interpret analysis instructions, operate on physiological signals, derive quantitative evidence, and translate the results into scientifically grounded conclusions. To systematically evaluate this broader capability, we introduce BrainBench, a unified benchmark for comprehensive, instruction-conditioned EEG understanding. It comprises four subsets—Foundational Analysis, Sleep Assessment, Neurocognitive Assessment, and Physiological Integration—covering 17 datasets, 172 tasks, and over 4K real-data instances. Given an instruction and EEG recordings with optional physiological signals, a system must perform the requested analysis and produce scientifically grounded, verifiable outputs. We evaluate 13 representative LLMs across more than 100K executions under two paradigms: autonomous code execution and structured agentic analysis. Results vary substantially across models, subsets, difficulty levels, and execution paradigms, demonstrating that EEG competence depends not only on the underlying model, but also on how it is operationalized. BrainBench advances the exploration of artificial intelligence systems that reason over neural signals and support scalable, reproducible discovery in neuroscience.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG Understanding, Large Language Models, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rXm1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BrainBench evaluates instruction-conditioned EEG analysis rather than only fixed-label decoding. It contains 172 tasks and 4,105 instances across 17 datasets, organized into Foundational Analysis, Sleep Assessment, Neurocognitive Assessment, and Physiological Integration. Systems produce reports and optional artifacts, which are parsed and scored using numerical, categorical, set, sequence, semantic, and artifact checks. The paper evaluates 13 LLMs through CodeAct and an extended BrainAgent, keeping instance inputs and evaluation references fixed. It analyzes aggregate scores, task difficulty, output types, within-task variability, and reasoning effort. Structured workflows generally improve performance, although some strong models perform better with autonomous coding, and the relative advantage varies substantially by domain.

Critical strengths
  1. A substantial benchmark for an important capability beyond classification. The 172-task inventory spans signal processing, quantitative sleep assessment, multimodal analysis, and report/artifact generation. The use of concrete recordings and executable references makes many outputs objectively checkable. The separation between answer extraction and correctness scoring is sensible, and Appendix B reports cross-review by three domain experts. This could become a useful shared resource for evaluating scientific assistants, particularly where success requires several dependent operations.
  2. The evaluation examines failures and variability rather than reporting only a leaderboard. The paper includes validation-unit breakdowns, within-task variability, a reasoning-effort study, repeated-run analysis, token usage, and detailed failed execution traces. The Sleep Assessment parser audit covers 10,250 extractions with 99.10% agreement (Appendix E.9). This is valuable evidence about extraction reliability within that subset, and the reported exceptions to BrainAgent's advantage appropriately prevent a simple claim that one execution paradigm always wins.
Critical weaknesses
  1. The scoring can reward agreement with hidden implementation choices rather than faithful completion of the visible scientific task. In FA-30-Instance1 (Appendix B.5, p. 24), the visible instruction asks the system to report the band with the highest mean PLV and briefly explain it. The hidden rubric nevertheless assigns 60% of the score to four separate numerical PLVs. Appendix C.2 requires unreported numbers to be parsed as null, and D.1 excludes intermediate calculations from scoring. A response satisfying the stated reporting requirement can therefore lose 60 points. The same example mandates theta as the unique semantic answer although theta and alpha differ by only about 0.00038, much less than their approximately 0.017 numerical tolerances. FA-01 also fixes Welch parameters in the reference script that are not fully specified in the visible instruction (p. 26). Expert cross-review and the parser audit do not establish that scientifically reasonable alternate implementations receive appropriate credit. This matters directly to the validity of the benchmark's central measurement.
  2. The comparative experiment changes more than execution organization, and the residual uncertainty is not sufficiently quantified. BrainAgent is substantially adapted to the benchmark's capability space, including an expanded toolset and revised planning/state mechanisms (Appendix D.2). It receives 32 GB memory versus CodeAct's 8 GB, and CodeAct additionally faces a 150-second per-execution limit, 20 interaction rounds, and a 1,024-token final-response cap (Tables 9–10). Appendix E.8 confirms higher token consumption by BrainAgent. These are legitimate system configurations, but holding inputs and scoring fixed does not isolate a general effect of structured versus autonomous execution. The paper needs to quantify how much of the advantage persists under comparable resources and helper access. In addition, only the first five source-ordered subjects per dataset are used, and parser agreement does not validate semantic/VLM judgments or quantify uncertainty from subjects and task templates. Consequently, the current evidence supports comparisons between these particular systems more strongly than broader claims about EEG understanding or cross-context reliability.
Questions for authors
  1. Clarification / scoring validity: Is the FA-30 instruction on p. 24 the complete instruction actually shown to systems? If so, how many other instances score outputs that are not explicitly requested? Please audit this mismatch and quantify its effect on scores or rankings. If additional reporting requirements were visible, please provide them and reconcile the example with the stated information boundary.
  2. Assessment-changing validation: How were numerical tolerances, spectral-analysis conventions, and near-tie rankings validated against independently implemented, scientifically acceptable analyses? For example, would an implementation choosing alpha rather than theta in the near-tied FA-30 example necessarily be scientifically wrong? Please report agreement with independent expert scoring of complete outputs, identify the parser/semantic/VLM judge models and auditor, and distinguish extraction fidelity from scientific correctness. The existing 99.10% parser agreement is relevant but answers a narrower question.
  3. Assessment-changing comparison: How do the principal BrainAgent–CodeAct gaps change when memory and execution budgets are matched, and when CodeAct can call the same reusable signal-analysis primitives? Please quantify memory/time/output-limit failures and provide uncertainty for representative score differences at the task or subject level. This would determine whether the main findings reflect workflow organization, resource allocation, tool access, or a combination. These requests concern validation of the current benchmark. Expanding to MEG/fMRI or collecting broader populations is future work and would not resolve the present instruction/scoring mismatch by itself.
9 comments
AuthorsOpenReview ↗

Response to Reviewer rXm1[1/N]

We thank the reviewer for recognizing BrainBench’s broad coverage of EEG analysis beyond classification and the value of our analyses of failures and variability. We appreciate your careful comments on alignment between instructions and scoring, scientific validation, and the comparability and uncertainty of system evaluations. We address each concern below.

Main Comments

W1: The scoring can reward agreement with hidden implementation choices rather than faithful completion of the visible scientific task. In FA-30-Instance1 (Appendix B.5, p. 24),...,FA-01 also fixes Welch parameters in the reference script,..., Expert cross-review and the parser audit do not establish that scientifically reasonable alternate implementations receive appropriate credit,...

A1: We sincerely thank the reviewer for the careful reading and highlighting these points. We address these concerns in detail below.

1. FA-01: Task design and validation of numerical tolerances

FA-01 also fixes Welch parameters in the reference script that are not fully specified in the visible instruction.

For numerical tasks, we considered how reasonable differences in execution could affect the requested result. Where the instruction left implementation choices open, we examined alternative filter choices, the order of filtering and data selection, referencing conventions, and other relevant settings, provided that they remained compatible with the intended analysis. Variation across these reference calculations informed the numerical acceptance criteria. For stable, non-zero reference values, we typically used a tolerance of approximately 5% of the reference magnitude. Where acceptable implementation choices produced greater variation, we used 10% or a larger task-specific tolerance where justified. When unspecified processing choices produced materially different answers, we made the relevant requirements explicit in the instruction. For example, we specified the filter type or preprocessing order when these choices substantially affected the result. This approach combines task-specific tolerances with precise instructions to reduce penalties arising from unstated reference conventions. For FA-01, reference validation kept the required 0.5–40 Hz FIR preprocessing fixed, while examining variations in Welch PSD estimation, including 2- and 4-second windows, different overlap ratios, and Hann and Hamming windows. Based on this validation, we set the relative tolerance for FA-01 to 10% to accommodate reasonable implementation differences consistent with the requested analysis. The fixed Welch parameters in the reference code represent one reference implementation. Agents may use other scientifically justified pipelines that follow the instruction, with their outputs assessed against the task’s numerical acceptance criteria.

2. FA-30: Clarifying reporting requirements and improving instance selection

The hidden rubric nevertheless assigns 60% of the score to four separate numerical PLVs.

The same example mandates theta as the unique semantic answer although theta and alpha differ by only about 0.00038, much less than their approximately 0.017 numerical tolerances.

We sincerely thank the reviewer for identifying this task-design oversight and apologize for the unclear reporting requirement. Our intended output was the global mean PLV for each of the four bands, followed by identification and interpretation of the dominant synchronization band. We have made this requirement explicit:

Report the global mean PLV for each of the delta, theta, alpha, and beta bands, identify the band with the highest value, and briefly explain the result.

For the cited instance, the four numerical PLVs remained within their acceptance tolerances across the tested implementations. However, our original instance selection did not adequately account for the sensitivity of a highest-band decision when the leading values are very close (many thanks for pointing this). The appropriate design is to select analysis windows with clearer separation and a stable ordering across the permitted implementations.

We therefore reviewed all 25 FA-30 instances and identified three with a relative gap between the two highest global mean PLV values: FA-30-01 (0.1108%), FA-30-03 (0.3265%), and FA-30-10 (0.0442%). We have replaced these three analysis windows with better-separated segments, verify their ranking stability across reasonable implementations, and update the corresponding instructions and reference targets. We are sincerely grateful to the reviewer for identifying this subtle but important aspect of task design that we had previously overlooked. Your careful observation has helped us further improve the rigor and reliability of our evaluation.

AuthorsOpenReview ↗

Response to Reviewer rXm1[2/N]

3. Accommodating scientifically reasonable alternative implementations

Expert cross-review and the parser audit do not establish that scientifically reasonable alternate implementations receive appropriate credit. This matters directly to the validity of the benchmark's central measurement.

Thank you for emphasizing this point. We explicitly consider reasonable methodological variation when constructing tasks and instances and defining their validation criteria. Through cross-expert review, we define the analytical objective and capability assessed by each task. Then, domain experts construct concrete instances, including the recordings, instructions, reference targets, and evaluation criteria. Two principles guide this process.

1. Accounting for implementation variability in numerical evaluation.

As detailed in our response under “FA-01: Task design and validation of numerical tolerances,” we examine instruction-compatible alternatives where methodological choices remain open and use the resulting variation to inform numerical tolerances. When unspecified choices materially change the result, the relevant methodological requirements need to be explicit. The intended target is the scientific analysis defined by the instruction, with appropriate allowance for implementation variation.

2. Designing tasks for stable comparison and meaningful interpretation.

When exact numerical agreement is not the appropriate target, we combine suitable validation units with carefully selected instances whose assessed comparisons remain stable across plausible methods.

SA-27 illustrates this principle. The instruction leaves the choice of EMG activity measure open. We selected three segments with clearly separated activity levels and verified consistent ordering across RMS, mean absolute amplitude, and mean-envelope measures. Sequence validation assesses the ordering, while semantic validation separately assesses REM identification. An agent therefore need not reproduce one particular feature formula or numerical scale to receive credit for the comparison.

The six validation units can be flexibly combined to assess the results, interpretations, and deliverables required by an instruction. Together with precise task specifications and stability-aware instance construction, this approach is intended to reward scientifically appropriate outcomes while reducing penalties caused solely by reasonable implementation differences. We hope this clarifies how methodological flexibility is incorporated into the benchmark’s design and evaluation.


W2: The comparative experiment changes more than execution organization, ... It receives 32 GB memory versus CodeAct's 8 GB, and CodeAct additionally faces a 150-second per-execution limit, 20 interaction rounds, and a 1,024-token final-response cap (Tables 9–10),..., The paper needs to quantify how much of the advantage persists under comparable resources and helper access. In addition, only the first five source-ordered subjects per dataset are used,..., Consequently, the current evidence supports comparisons between these particular systems more strongly than broader claims about EEG understanding or cross-context reliability.

A2: Many thanks for your valuable comments. We clarify the execution controls, the purpose of the system comparison, and the scope of the supporting evidence below.

1. Execution limits prevent unproductive execution and keep evaluation costs manageable

CodeAct additionally faces a 150-second per-execution limit, 20 interaction rounds, and a 1,024-token final-response cap.

We introduced these execution limits primarily to prevent unproductive behavior in CodeAct, including unbounded cycles of repeated code execution and interaction following execution errors, prolonged retries after API failures, and repetitive generation. These limits help ensure reproducible evaluation while keeping evaluation costs within a reasonable budget. BrainAgent is likewise subject to limits on specialist-agent planning and tool execution. We have explicitly documented these controls in the Appendix D.2.4.

AuthorsOpenReview ↗

Response to Reviewer rXm1[3/N]

2. Benchmark scope and additional evaluation with shared analytical tools

The paper needs to quantify how much of the advantage persists under comparable resources and helper access.

Following the your suggestion, we additionally evaluated CodeAct under the same Docker configuration as BrainAgent and with access to the same reusable tools, while retaining CodeAct’s autonomous execution workflow. This configuration is denoted as CodeAct+Tools. Using Qwen3.7 Plus and Qwen3.7 Max, we compared original CodeAct, original BrainAgent, and CodeAct+Tools on the FA subset. The results are summarized below.

MetricQwen3.7 PlusQwen3.7 Max
Original CodeAct score68.3369.54
Original BrainAgent score73.0872.21
CodeAct+Tools score69.1564.34
Shared-tool coverage on non-knowledge instances99.00%98.44%
Mean interaction rounds, CodeAct → CodeAct+Tools8.48 → 10.915.09 → 13.77
Repeated-failure terminations, CodeAct → CodeAct+Tools1 → 402 → 37
Mean target-model tokens, CodeAct → CodeAct+Tools21,320 → 45,4558,965 → 41,622

Shared tools were invoked in more than 98% of non-knowledge instances for both models, indicating that they were widely used during evaluation. However, shared-tool access did not consistently improve performance or eliminate the observed gap between the two systems. Both models also required more interaction rounds and target-model tokens after receiving tool access, while repeated-failure terminations increased from 1 to 40 for Plus and from 2 to 37 for Max. These results suggest that access to analytical tools alone does not guarantee better task performance or more efficient execution; effectively incorporating those tools into the execution workflow also matters. The supplementary experiment thus provides evidence about both the performance and execution costs associated with adding shared tools to CodeAct.

3. Semantic/VLM scoring, subject coverage, and consistency analyses

Only the first five source-ordered subjects per dataset are used, and parser agreement does not validate semantic/VLM judgments or quantify uncertainty from subjects and task templates.

Thank you for raising these valuable questions. We clarify our scoring procedure, subject selection, and existing consistency analyses below.

1. Semantic and VLM assessment follows explicit scoring instructions.

For semantic and visual assessment, we provide detailed scoring prompts specifying how the requested content and visual outputs should be evaluated, and use Qwen3.7-Max as a strong judge model. This assessment procedure is distinct from the extraction process examined in the parser audit.

2. The five-subject design balances recording diversity, analytical coverage, and evaluation cost.

BrainBench evaluates instruction-following and analytical execution by fixed LLM-based systems, rather than cross-subject predictive generalization in trained EEG decoders. Within each dataset, the common acquisition paradigm and recording setup allow the same analytical capabilities to be examined across multiple subjects under consistent acquisition conditions.

Selecting five subjects in a fixed source order provides reproducible, multi-subject coverage while keeping evaluation across datasets and tasks computationally manageable. Including every available subject would substantially increase repeated execution within the same dataset and task settings without necessarily expanding the analytical capabilities assessed. This design balances recording diversity with broader coverage of datasets and tasks.

3. Task-level uncertainty analysis.

Following your suggestion, we have quantified task-level uncertainty in the BrainAgent–CodeAct score differences, as detailed in our response to Q3.

Reviewer_xmcqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BrainBench evaluates instruction-conditioned EEG analysis through 172 tasks and 4,105 instances drawn from 17 datasets, covering foundational analysis, sleep assessment, neurocognitive assessment, and physiological integration. It compares 13 LLMs using autonomous code execution through CodeAct and structured analytical tools through BrainAgent. Outputs are evaluated using numerical, categorical, set, sequence, semantic, and artifact checks against task-specific references.

The paper finds that structured tools improve overall scores for most models, although the strongest overall result comes from CodeAct. It also analyzes performance across difficulty categories, reasoning effort, execution consistency, and failure cases. Its main contribution is a broad benchmark of analytical workflows beyond conventional EEG decoding.

Critical strengths
  • Broad and useful workflow coverage. The benchmark includes preprocessing, quantitative analysis, multimodal integration, artifact generation, and invalid-request handling. This provides a richer basis for evaluating EEG assistants than fixed-label prediction alone.
  • Informative empirical comparisons and traces. Testing multiple models under two execution frameworks, alongside reasoning and failure analyses, exposes meaningful differences in system behavior. The traces make concrete failures visible that aggregate scores would obscure.
Critical weaknesses
  • What the benchmark measures is not sufficiently disentangled. The paper describes deterministic reference scripts but does not clearly document their source, validation, or possible implementation overlap with BrainAgent tools. Some tools already perform the main analytical operation, such as respiratory-event detection. It is therefore difficult to distinguish scientific analytical competence from selecting supplied implementations and matching reference outputs. Although there are artifact and invalid-request tasks, the paper does not establish systematic evaluation of whether agents recognize when a recording needs different preprocessing or when an analysis is unjustified. This limits the interpretation of “comprehensive EEG understanding” which is being evaluated.
  • Difficulty is insufficiently operationalized for its role in the findings and scores. The easy/medium/hard categories are qualitative, without a clearly reported assignment procedure or difficulty-specific expert agreement. They also do not distinguish the complexity of the underlying analysis from the work remaining once specialized tools are available. This is an important point to clarify as many of the benchmark's findings depend this definition.
Questions for authors
  • Reference source and validation: How were the reference scripts written? How were these reference scripts validated for correctness of scientific analysis? Do the reference scripts share code or analytical defaults with BrainAgent’s tools?
  • Benchmark evaluation of analysis: Does the benchmark reward agents for recognizing artifacts or poor signal quality and appropriately modifying the analysis—for example, excluding unsuitable channels or declining an unsupported conclusion? If an agent chooses a scientifically defensible method that differs from the reference pipeline, can it still receive credit?
  • Difficulty assignment: How were difficulty levels assigned and validated? Do they correspond to the number of analytical steps or tool calls required, given the tools available to the agent? How is difficulty assessed when a single tool performs most of the analysis?
10 comments
AuthorsOpenReview ↗

Response to Reviewer xmcq[1/N]

We thank you for recognizing BrainBench’s broad workflow coverage and the value of our empirical comparisons and execution traces. We appreciate your constructive comments on reference construction and validation, the analytical capabilities assessed, and task difficulty. We clarify these aspects of the evaluation protocol and address each concern below.

Main Comments

W1.1: ... The paper describes deterministic reference scripts but does not clearly document their source, validation, or possible implementation overlap with BrainAgent tools. ... It is therefore difficult to distinguish scientific analytical competence from selecting supplied implementations and matching reference outputs.

A1.1: Thank you for raising these important questions. We address your concerns from the following perspectives.

1. Construction and validation of reference scripts

The paper describes deterministic reference scripts but does not clearly document their source, validation

Our reference scripts were primarily implemented using MNE-Python and related scientific Python libraries. Each task was decomposed into explicit processing and calculation steps. We consulted publicly available EEG-processing implementations on GitHub as implementation references and used an OpenAI flagship model to assist with code drafting. The resulting scripts were manually reviewed for analytical logic, parameter settings, and consistency among the instruction, requested outputs, and scoring configuration.
We distinguish the correctness of reference implementations from the robustness of the evaluation criteria. Reference implementations were reviewed against the intended analytical definitions and, where applicable, checked using alternative calculations or dataset annotations. We further examined whether reasonable implementation choices could change the expected outputs and designed task-specific validation criteria accordingly. We illustrate these considerations below.

  1. Accounting for implementation variability in numerical evaluation.

For numerical tasks, we considered how reasonable differences in execution could affect the requested result. Where the instruction left implementation choices open, we examined alternative filter choices, the order of filtering and data selection, referencing conventions, etc., provided that they remained compatible with the intended analysis. The variation across these reference calculations informed the numerical acceptance criteria. For stable, non-zero reference values, we typically used a tolerance of approximately 5% of the reference magnitude. Where acceptable implementation choices produced greater variation, we used 10% or a larger task-specific tolerance where justified.

When unspecified processing choices produced materially different answers, we made the relevant requirements explicit in the instruction. For example, we specified the filter type or preprocessing order when these choices substantially affected the result. This approach combines task-specific tolerances with sufficiently precise instructions, reducing the risk that an agent follows a reasonable interpretation but is penalized for differing from an unstated reference convention.|

AuthorsOpenReview ↗

Response to Reviewer xmcq[2/N]

  1. Designing tasks for stable comparison and meaningful interpretation.

To better illustrate the principles underlying our task design, we take SA-27 as an example and explain the key considerations behind its construction.

SA-27: The agent compares EMG activity across three one-minute sleep segments, ranks them in descending order, and determines whether REM sleep is present and, if so, which segment corresponds to REM.

The instruction deliberately leaves the choice of EMG activity measure open. The agent can select an appropriate method for quantifying EMG strength without being required to reproduce a single prescribed feature formula. Because this component is evaluated through sequence validation, the assessment focuses on the resulting comparison rather than exact agreement with one implementation’s numerical values. This preserves the agent’s freedom to choose how to perform the analysis while maintaining a clearly defined evaluation target.

To support this flexibility, we selected segments with clearly separated EMG activity levels. The construction script checks that RMS, mean absolute amplitude, and mean EMG-envelope measures produce the same ordering, together with sufficient separation between adjacent segments. Thus, the choice of evaluation instances is informed by multiple plausible measures, reducing the risk that small implementation differences change the expected ranking.

The instances include both REM-present cases, in which the lowest-EMG segment is REM, and REM-absent cases, in which the lowest-EMG segment is not REM, as determined from the sleep-stage labels. The segment-label ordering also varies across instances. Consequently, neither a fixed ordering nor automatically assigning REM to the lowest-EMG segment is consistently sufficient. Sequence validation contributes 60 points for the EMG ordering, and semantic consistency contributes 40 points for correctly stating REM presence or absence and identifying the relevant segment when applicable. This design preserves methodological freedom in the analysis while separately evaluating the quantitative comparison and the correctness of the physiological interpretation.

Consequently, each task and its reference script were designed and reviewed together, with deliberate attention to implementation variability, ambiguous outputs, unstable comparisons, and interpretive shortcuts that could arise during actual agent evaluation.

2. Task-oriented references and capability-oriented tools

possible implementation overlap with BrainAgent tools.

Our reference scripts are task-oriented, while BrainAgent’s tools are capability-oriented. The scripts implement offline reference calculations independently of BrainAgent’s tool-execution interfaces, with attention to implementation correctness and evaluation robustness. The tools provide configurable operations for use across tasks. Both may share standard scientific libraries and analytical definitions, but the tools do not embed individual benchmark answers.

SA-27 illustrates this distinction. Its reference script implements the specified task and establishes the expected EMG ordering and REM status. By comparison, EMGActivityAnalyzer exposes nine input parameters through the agent harness to provide general-purpose EMG activity analysis, an important component of sleep assessment. It is designed as a reusable analytical capability across recordings and analysis requests, rather than a tool dedicated to a particular benchmark task.

This example illustrates how task-specific reference construction and reusable analytical capabilities serve distinct roles within the benchmark.

AuthorsOpenReview ↗

Response to Reviewer xmcq[3/N]

3. Assessing analytical competence across complete workflows

Some tools already perform the main analytical operation, such as respiratory-event detection. It is therefore difficult to distinguish scientific analytical competence from selecting supplied implementations and matching reference outputs.

A tool may perform a key computation, but its output alone does not establish that the complete analytical task has been fulfilled. Our task design includes requirements for preparing inputs, configuring analytical operations, combining intermediate results, and reaching conclusions consistent with the instruction.

Importantly, the analytical competence assessed by BrainBench extends beyond implementing individual signal-processing operations. It includes translating scientific questions into appropriate analytical procedures, configuring and coordinating the required operations, integrating intermediate evidence, and drawing physiologically grounded conclusions. To further illustrate how these considerations are reflected in our task design, we provide two representative examples below.

  • SA-27: The system must prepare the recording, construct the three specified signal segments, obtain EMG measurements for each segment, compare and rank those measurements, and determine whether REM sleep is present. EMGActivityAnalyzer only performs the computation specified through its interface for the supplied signal input. It does not rank A/B/C, identify the lowest-EMG segment, or determine REM status. These comparisons and judgments must be organized and completed by the agent using the available analytical results and physiological evidence.

  • SA-36: This is a Hard task that requires the agent to analyze an entire overnight sleep recording and calculate the apnea–hypopnea indices separately for REM and NREM sleep (REM-AHI and NREM-AHI) by integrating respiratory signals with sleep-stage information. The system must obtain the relevant respiratory events, associate them with sleep stages according to their onset times, exclude Wake-onset events, calculate the corresponding REM and NREM durations, and produce the two stage-specific indices. The agent must translate these analytical requirements into appropriate tool configurations and data dependencies, and organize the operations into a coherent workflow.

    This type of compositional analysis is particularly challenging, as it requires extensive matching and validation between discrete sleep-stage annotations and respiratory events. Across the 13 models evaluated under both CodeAct and BrainAgent execution paradigms, the mean SA-36 score remained below 10 out of 100.

These examples demonstrate that access to specialized analytical tools does not, by itself, guarantee successful completion of scientific analysis tasks. BrainBench evaluates whether an agent, operating through either autonomous code execution or a structured workflow, can translate scientific requirements into executable workflows, appropriately integrate analytical results, and produce conclusions supported by physiological evidence. The benchmark therefore evaluates end-to-end analytical competence at the system level, through the correctness and consistency of the final reports and requested artifacts.

Reviewer_kKjPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates how well LLMs can follow instructions and perform EEG analyses on real physiological recordings. It contains 172 tasks and 4105 instances from 17 datasets, and covers Foundational Analysis, Sleep Assessment, Neurocognitive Assessment, and Physiological Integration. This work tests 13 LLMs with two existing frameworks, CodeAct and BrainAgent, using the same instructions, inputs, and scoring criteria. Six validation units are used to score different kinds of output. The experiments evaluate performance across domains, difficulty levels, output types, and execution settings. Overall, the paper is well structured and performs a broad range of experiments.

Critical strengths
  1. The topic is of interest. It tries to examine whether LLMs can follow analysis instructions, process real EEG recordings, and generate reports, providing insights into their capabilities and limitations in EEG analysis and interpretation.
  2. The experiment is relatively comprehensive. It covers various tasks, datasets, and evaluation models.
Critical weaknesses
  1. While construction of EEG benchmark and its empirical evaluation have practical values, the methodological novelty and new insights beyond existing evaluations are limited.
  2. The choice and organization of the four subsets require further justification. In section 3.1, it explains what each subset evaluates. However, the rationale for why these four subsets represent comprehensive EEG understanding remains unclear. Additionally, the organization combines general analytical operations, application domains, and multimodal integration, while these dimensions may overlap.
  3. Although BrainBench contains a large number of instances across multiple datasets, it includes only five subjects per dataset, and multiple instances reuse the same recordings.
  4. CodeAct and BrainAgent adopt different domain-specific tools and workflows. It would be more convincing to control these factors for comparison.
Questions for authors

see above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer kKjP[1/N]

We thank you for recognizing the relevance of our research question, the clarity of the paper, and the breadth of our experiments. We appreciate your constructive comments on the benchmark’s contributions and organization, subject coverage, and comparisons between execution systems. We clarify these points and address each concern below.

Main Comments

Q1: While construction of EEG benchmark and its empirical evaluation have practical values, the methodological novelty and new insights beyond existing evaluations are limited.

A1: Thank you for your comments. We clarify our contributions and the resulting insights below.

BrainBench addresses a critical gap in the systematic evaluation of LLM-based EEG agents. As increasingly capable EEG agents emerge, their evaluations remain fragmented across different tasks, datasets, output requirements, and success criteria. This makes it difficult for researchers to compare systems, identify their limitations, and measure progress. BrainBench establishes a shared evaluation framework spanning 172 tasks and 17 datasets. Our central contribution is to provide the community with a reusable methodology and benchmark for assessing EEG agents across heterogeneous analytical workflows.

1. A natural-language interface designed for community reuse, supported by compositional evaluation.

The choice of natural-language input and output is central to making the benchmark reusable by emerging EEG agents. These systems may differ substantially in their tools, planning mechanisms, and internal representations, but natural-language interaction provides a common external interface. BrainBench therefore accepts natural-language instructions together with recordings and evaluates the resulting natural-language reports and requested artifacts. Researchers can connect different agents through this interface while retaining their own execution architectures, reducing the need for system-specific evaluation procedures or a single prescribed response format.

This interface also reflects how EEG assistants communicate their findings to researchers. A correct numerical result does not necessarily imply a correct interpretation. An agent may calculate a measurement accurately while drawing an inconsistent conclusion or making claims unsupported by the available evidence.

To make these heterogeneous reports assessable, BrainBench combines numerical, categorical, set, sequence, semantic, and artifact validation (Sec. 3.2.3 and Appendix C). The parser extracts explicitly reported information, and the corresponding validation units assess its correctness against task-specific references and criteria. Their composition allows a single task to evaluate quantitative results, the consistency of the accompanying explanation, unsupported claims, and the required deliverables. This connects a flexible, community-facing interface with a common and verifiable evaluation protocol.

2. A common EEG benchmark for autonomous code execution and structured-agent execution.

Building on executable-code evaluations such as HEARTS [1], BrainBench systematically evaluates the same LLMs through both autonomous coding and structured EEG-agent workflows under shared analytical instructions, recordings, and assessment criteria.

The added value is that researchers can examine how a model’s EEG capabilities are expressed through different execution systems within the same task space. A model may be effective at independently implementing an analysis, at coordinating reusable analytical tools, or at different combinations of these capabilities. Evaluating both settings makes these differences visible and broadens the benchmark’s relevance to emerging EEG agents with different designs. The protocol and implementations are described in Sec. 3.2.4 and Appendix D.

AuthorsOpenReview ↗

Response to Reviewer kKjP[2/N]

3. Insights that inform practical decisions in EEG-agent development.

Beyond an overall leaderboard, our evaluation addresses three questions that directly concern researchers developing and deploying EEG agents:

  • How should researchers choose a model and execution configuration for different EEG applications and levels of analytical difficulty? Our analyses characterize how relative strengths and limitations change across domains and task complexity, supporting application- and difficulty-aware system selection rather than reliance on a single overall ranking (Sec.4.1–4.2 and Appendix E.2).

  • Which capabilities benefit from structured-agent execution or autonomous coding? By examining validation-unit performance and consistency across recordings, BrainBench distinguishes improvements in quantitative analysis from changes in interpretation, deliverable quality, and execution consistency. This reveals why gains in one component cannot be assumed to extend to the entire analytical workflow (Sec. 4.3–4.4 and Appendices E.3–E.4).

  • When is additional reasoning worth its computational cost? Our joint analysis of performance and token consumption examines the returns from increasing reasoning effort across tasks and execution systems. It provides evidence for deciding when additional inference-time computation is useful and when its returns diminish (Sec. 4.5, Appendix E.5, and Fig. 12).

These contributions make BrainBench a shared basis for evaluating new EEG agents and for making evidence-based choices about their models, execution systems, and reasoning budgets.

We thank the reviewer again and hope this clarifies the methodological contribution and the practical value of the insights enabled by the benchmark.

[1] Sirui Li et al. HEARTS: Benchmarking LLM Reasoning on Health Time Series. ICML 2026. Paper


Q2: The choice and organization of the four subsets require further justification. In section 3.1, it explains what each subset evaluates. However, the rationale for why these four subsets represent comprehensive EEG understanding remains unclear. Additionally, the organization combines general analytical operations, application domains, and multimodal integration, while these dimensions may overlap.

A2: Thank you for raising this point. We clarify the rationale for selecting the four subsets and the intended scope of “comprehensive EEG understanding” below.

1. The subsets were selected to serve recurring research needs across the EEG community.

BrainBench is intended to provide a shared evaluation resource for researchers developing EEG agents for different analytical purposes. Our organizing principle is broad applicability to established EEG research workflows. General signal processing and feature analysis provide capabilities needed across applications. Sleep and neurocognitive research provide widely studied settings with different physiological questions and analytical demands. Joint analysis with other physiological signals extends this coverage to workflows in which EEG must be interpreted alongside complementary evidence.

These settings allow the benchmark to serve both researchers developing general EEG analysis methods and those building agents for specific applications. Their selection therefore reflects the combination of shared analytical foundations, established application needs, and integration across physiological measurements.

The overlap noted by the reviewer follows from this application-oriented organization. An analytical operation may be evaluated on its own and also used within a broader research workflow. Retaining both settings allows us to assess whether an agent can perform an operation and whether it can use that operation appropriately to answer a physiological question. The subsets consequently organize complementary evaluation scenarios; they are not intended to form a mutually exclusive taxonomy of EEG methods.

We recognize that these four subsets do not encompass every EEG application. As research questions and experimental paradigms evolve, additional scenarios can be incorporated through the same task and evaluation framework. The present benchmark provides a broadly applicable foundation that the community can reuse and extend.

AuthorsOpenReview ↗

Response to Reviewer kKjP[3/N]

2. “Comprehensive” concerns the breadth of analytical capabilities and their assessment, including the outputs an EEG agent must deliver.

The four subsets provide coverage of research scenarios, while the six complementary validation units provide coverage of the different requirements of an analytical response. BrainBench evaluates numerical results, discrete decisions, collections and ordered outputs, natural-language interpretations, and generated artifacts. Task-specific combinations allow analytical results and their accompanying explanations or deliverables to be evaluated together, including whether the reported conclusions are consistent with the reference evidence and the instruction.

This breadth provides the community with a reusable benchmark for assessing and advancing EEG agents across diverse analytical settings. Through a shared natural-language interface, researchers can compare agents with different implementations against common requirements and extend the evaluation to additional tasks.

To make the intended meaning of “comprehensive EEG understanding” explicit, we have clarified the definition in Sec. 3.1 as follows:

In BrainBench, comprehensive EEG understanding refers to the breadth of EEG analysis capabilities across common research workflows. We assess these capabilities through instruction-conditioned analytical execution, evaluating quantitative results, structured decisions, physiological interpretations, and requested artifacts.

We sincerely thank the reviewer for this valuable concern. We hope our clarification adequately addresses your concerns.


Q3: Although BrainBench contains a large number of instances across multiple datasets, it includes only five subjects per dataset, and multiple instances reuse the same recordings.

A3: Thank you for raising these questions about subject coverage and recording reuse. BrainBench evaluates whether a fixed LLM-based system can interpret an instruction, execute the required analysis, and report a supported conclusion. The subject selection balances repeated evaluation against execution cost, while recording reuse provides a common data context for testing different analytical capabilities. We address the two concerns separately.

it includes only five subjects per dataset
  1. Our evaluation target differs from the cross-subject predictive generalization of a trained EEG decoder. For a given task, recordings from the same dataset share an acquisition and file-format context, and the system applies the same analytical specification to different signal realizations. Increasing the subject count therefore primarily adds repetitions of an existing type of analysis. Five subjects provide a basic test of whether the workflow can be applied across individuals, with reference outputs computed separately for each recording. This enables repeated evaluation beyond a single subject without requiring every task to be executed on the entire source cohort.

  2. Each additional subject requires complete agent executions across the applicable tasks and models. For example, expanding from 5 to all 100 subjects in ISRUC-Sleep Subgroup I would require 20 times as many subject-dependent task executions under fixed tasks, models, execution settings, and instances per subject. The associated execution cost would increase approximately twentyfold if average cost per run remained comparable. This expense recurs whenever researchers evaluate a new model or agent. Selecting five subjects therefore retains repeated testing across individuals while keeping BrainBench practical for both our experiments and subsequent users.

multiple instances reuse the same recordings

Reusing recordings helps control input-related variability when evaluating different analytical tasks. If every task used different recordings, differences in subject characteristics, signal quality, and acquisition conditions would introduce additional variation into task comparisons. Applying multiple tasks to a shared set of recordings provides a common physiological context, helping distinguish differences in analytical requirements from differences in the input data.

The reused recording does not make the tasks equivalent. Spectral estimation, channel comparison, and connectivity analysis require different operations and produce different outputs, even when performed on the same signal. A single recording can therefore support several distinct tests of analytical competence. Each task–recording pair has its own instruction, reference outputs, and evaluation requirements. Recording reuse expands the range of capabilities evaluated without treating these instances as additional independent subjects.

We thank the reviewer again and hope this clarifies the distinct purposes of subject selection and recording reuse.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank the reviewers for their careful reading and constructive feedback. All three reviewers (kKjP, xmcq, and rXm1) recognize BrainBench's broad coverage and the practical value of evaluating LLM-based EEG analysis. The reviews also highlight our informative comparisons, execution traces, and analyses of failures and variability.

The main concerns involve benchmark scope, reference and scoring validity, difficulty assignment, and system comparisons. We have responded through manuscript revisions, benchmark corrections, and additional validation and robustness analyses. Below, we summarize these updates and their locations in the revised manuscript or our point-by-point responses.

  1. Benchmark scope and community reuse. In response to the suggestions of Reviewers kKjP, xmcq, and rXm1, we clarified the benchmark's scope, subset rationale, subject selection, recording reuse, and natural-language interface for evaluating diverse EEG agents. (Sections 3.1 and 3.2.4, pp. 4 and 6; Appendix A, p. 16.)

  2. Difficulty assignment and weighting robustness. In response to the suggestions of Reviewer xmcq, we documented expert-led difficulty assignment before model evaluation and provided an unweighted scoring analysis in our point-by-point response. (Appendix B, p. 16.)

  3. Reference validation and methodological flexibility. In response to the suggestions of Reviewers xmcq and rXm1, we clarified reference-script construction, expert review, and task-design principles that accommodate scientifically justified implementation choices. (Appendix C.1, pp. 27–28.)

  4. Instruction–scoring alignment. In response to Reviewer rXm1's feedback on FA-30, we clarified its reporting requirements and replaced three implementation-sensitive instances with verified stable ones. (Revised FA-30 example in Appendix B.5, p. 25.)

  5. Judge disclosure and expert review. In response to the suggestions of Reviewer rXm1, we disclosed the evaluator models and provided an additional expert audit of semantic scoring in our point-by-point response. (Evaluator configuration in Appendix D.1, p. 31.)

  6. Shared-tool evaluation. In response to the suggestions of Reviewers kKjP and rXm1, we added evaluations giving CodeAct access to reusable BrainAgent tools and analyzed performance and execution costs in our point-by-point responses.

  7. Uncertainty and execution diagnostics. In response to the suggestions of Reviewer rXm1, we added task-bootstrap confidence intervals and execution diagnostics to characterize uncertainty in system comparisons and distinguish execution failures from analytical correctness. (Appendices E.9–E.10, p. 48.)

We hope that BrainBench contributes to advancing LLM-based EEG analysis by establishing a systematic and rigorous evaluation framework across diverse EEG tasks, identifying critical capability gaps, and guiding the development of more reliable, generalizable, and scientifically grounded intelligent systems for EEG research.

We sincerely welcome the reviewers' further feedback and constructive insights to refine and strengthen our work, and to help establish BrainBench as a valuable and reusable resource for the broader EEG research community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.