- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BrainBench evaluates instruction-conditioned EEG analysis rather than only fixed-label decoding. It contains 172 tasks and 4,105 instances across 17 datasets, organized into Foundational Analysis, Sleep Assessment, Neurocognitive Assessment, and Physiological Integration. Systems produce reports and optional artifacts, which are parsed and scored using numerical, categorical, set, sequence, semantic, and artifact checks. The paper evaluates 13 LLMs through CodeAct and an extended BrainAgent, keeping instance inputs and evaluation references fixed. It analyzes aggregate scores, task difficulty, output types, within-task variability, and reasoning effort. Structured workflows generally improve performance, although some strong models perform better with autonomous coding, and the relative advantage varies substantially by domain.
Critical strengths
- A substantial benchmark for an important capability beyond classification. The 172-task inventory spans signal processing, quantitative sleep assessment, multimodal analysis, and report/artifact generation. The use of concrete recordings and executable references makes many outputs objectively checkable. The separation between answer extraction and correctness scoring is sensible, and Appendix B reports cross-review by three domain experts. This could become a useful shared resource for evaluating scientific assistants, particularly where success requires several dependent operations.
- The evaluation examines failures and variability rather than reporting only a leaderboard. The paper includes validation-unit breakdowns, within-task variability, a reasoning-effort study, repeated-run analysis, token usage, and detailed failed execution traces. The Sleep Assessment parser audit covers 10,250 extractions with 99.10% agreement (Appendix E.9). This is valuable evidence about extraction reliability within that subset, and the reported exceptions to BrainAgent's advantage appropriately prevent a simple claim that one execution paradigm always wins.
Critical weaknesses
- The scoring can reward agreement with hidden implementation choices rather than faithful completion of the visible scientific task. In FA-30-Instance1 (Appendix B.5, p. 24), the visible instruction asks the system to report the band with the highest mean PLV and briefly explain it. The hidden rubric nevertheless assigns 60% of the score to four separate numerical PLVs. Appendix C.2 requires unreported numbers to be parsed as null, and D.1 excludes intermediate calculations from scoring. A response satisfying the stated reporting requirement can therefore lose 60 points. The same example mandates theta as the unique semantic answer although theta and alpha differ by only about 0.00038, much less than their approximately 0.017 numerical tolerances. FA-01 also fixes Welch parameters in the reference script that are not fully specified in the visible instruction (p. 26). Expert cross-review and the parser audit do not establish that scientifically reasonable alternate implementations receive appropriate credit. This matters directly to the validity of the benchmark's central measurement.
- The comparative experiment changes more than execution organization, and the residual uncertainty is not sufficiently quantified. BrainAgent is substantially adapted to the benchmark's capability space, including an expanded toolset and revised planning/state mechanisms (Appendix D.2). It receives 32 GB memory versus CodeAct's 8 GB, and CodeAct additionally faces a 150-second per-execution limit, 20 interaction rounds, and a 1,024-token final-response cap (Tables 9–10). Appendix E.8 confirms higher token consumption by BrainAgent. These are legitimate system configurations, but holding inputs and scoring fixed does not isolate a general effect of structured versus autonomous execution. The paper needs to quantify how much of the advantage persists under comparable resources and helper access. In addition, only the first five source-ordered subjects per dataset are used, and parser agreement does not validate semantic/VLM judgments or quantify uncertainty from subjects and task templates. Consequently, the current evidence supports comparisons between these particular systems more strongly than broader claims about EEG understanding or cross-context reliability.
Questions for authors
- Clarification / scoring validity: Is the FA-30 instruction on p. 24 the complete instruction actually shown to systems? If so, how many other instances score outputs that are not explicitly requested? Please audit this mismatch and quantify its effect on scores or rankings. If additional reporting requirements were visible, please provide them and reconcile the example with the stated information boundary.
- Assessment-changing validation: How were numerical tolerances, spectral-analysis conventions, and near-tie rankings validated against independently implemented, scientifically acceptable analyses? For example, would an implementation choosing alpha rather than theta in the near-tied FA-30 example necessarily be scientifically wrong? Please report agreement with independent expert scoring of complete outputs, identify the parser/semantic/VLM judge models and auditor, and distinguish extraction fidelity from scientific correctness. The existing 99.10% parser agreement is relevant but answers a narrower question.
- Assessment-changing comparison: How do the principal BrainAgent–CodeAct gaps change when memory and execution budgets are matched, and when CodeAct can call the same reusable signal-analysis primitives? Please quantify memory/time/output-limit failures and provide uncertainty for representative score differences at the task or subject level. This would determine whether the main findings reflect workflow organization, resource allocation, tool access, or a combination. These requests concern validation of the current benchmark. Expanding to MEG/fMRI or collecting broader populations is future work and would not resolve the present instruction/scoring mismatch by itself.
9 comments
Response to Reviewer rXm1[2/N]
3. Accommodating scientifically reasonable alternative implementations
Expert cross-review and the parser audit do not establish that scientifically reasonable alternate implementations receive appropriate credit. This matters directly to the validity of the benchmark's central measurement.
Thank you for emphasizing this point. We explicitly consider reasonable methodological variation when constructing tasks and instances and defining their validation criteria. Through cross-expert review, we define the analytical objective and capability assessed by each task. Then, domain experts construct concrete instances, including the recordings, instructions, reference targets, and evaluation criteria. Two principles guide this process.
1. Accounting for implementation variability in numerical evaluation.
As detailed in our response under “FA-01: Task design and validation of numerical tolerances,” we examine instruction-compatible alternatives where methodological choices remain open and use the resulting variation to inform numerical tolerances. When unspecified choices materially change the result, the relevant methodological requirements need to be explicit. The intended target is the scientific analysis defined by the instruction, with appropriate allowance for implementation variation.
2. Designing tasks for stable comparison and meaningful interpretation.
When exact numerical agreement is not the appropriate target, we combine suitable validation units with carefully selected instances whose assessed comparisons remain stable across plausible methods.
SA-27 illustrates this principle. The instruction leaves the choice of EMG activity measure open. We selected three segments with clearly separated activity levels and verified consistent ordering across RMS, mean absolute amplitude, and mean-envelope measures. Sequence validation assesses the ordering, while semantic validation separately assesses REM identification. An agent therefore need not reproduce one particular feature formula or numerical scale to receive credit for the comparison.
The six validation units can be flexibly combined to assess the results, interpretations, and deliverables required by an instruction. Together with precise task specifications and stability-aware instance construction, this approach is intended to reward scientifically appropriate outcomes while reducing penalties caused solely by reasonable implementation differences. We hope this clarifies how methodological flexibility is incorporated into the benchmark’s design and evaluation.
W2: The comparative experiment changes more than execution organization, ... It receives 32 GB memory versus CodeAct's 8 GB, and CodeAct additionally faces a 150-second per-execution limit, 20 interaction rounds, and a 1,024-token final-response cap (Tables 9–10),..., The paper needs to quantify how much of the advantage persists under comparable resources and helper access. In addition, only the first five source-ordered subjects per dataset are used,..., Consequently, the current evidence supports comparisons between these particular systems more strongly than broader claims about EEG understanding or cross-context reliability.
A2: Many thanks for your valuable comments. We clarify the execution controls, the purpose of the system comparison, and the scope of the supporting evidence below.
1. Execution limits prevent unproductive execution and keep evaluation costs manageable
CodeAct additionally faces a 150-second per-execution limit, 20 interaction rounds, and a 1,024-token final-response cap.
We introduced these execution limits primarily to prevent unproductive behavior in CodeAct, including unbounded cycles of repeated code execution and interaction following execution errors, prolonged retries after API failures, and repetitive generation. These limits help ensure reproducible evaluation while keeping evaluation costs within a reasonable budget. BrainAgent is likewise subject to limits on specialist-agent planning and tool execution. We have explicitly documented these controls in the Appendix D.2.4.
Response to Reviewer rXm1[3/N]
2. Benchmark scope and additional evaluation with shared analytical tools
The paper needs to quantify how much of the advantage persists under comparable resources and helper access.
Following the your suggestion, we additionally evaluated CodeAct under the same Docker configuration as BrainAgent and with access to the same reusable tools, while retaining CodeAct’s autonomous execution workflow. This configuration is denoted as CodeAct+Tools. Using Qwen3.7 Plus and Qwen3.7 Max, we compared original CodeAct, original BrainAgent, and CodeAct+Tools on the FA subset. The results are summarized below.
| Metric | Qwen3.7 Plus | Qwen3.7 Max |
|---|---|---|
| Original CodeAct score | 68.33 | 69.54 |
| Original BrainAgent score | 73.08 | 72.21 |
| CodeAct+Tools score | 69.15 | 64.34 |
| Shared-tool coverage on non-knowledge instances | 99.00% | 98.44% |
| Mean interaction rounds, CodeAct → CodeAct+Tools | 8.48 → 10.91 | 5.09 → 13.77 |
| Repeated-failure terminations, CodeAct → CodeAct+Tools | 1 → 40 | 2 → 37 |
| Mean target-model tokens, CodeAct → CodeAct+Tools | 21,320 → 45,455 | 8,965 → 41,622 |
Shared tools were invoked in more than 98% of non-knowledge instances for both models, indicating that they were widely used during evaluation. However, shared-tool access did not consistently improve performance or eliminate the observed gap between the two systems. Both models also required more interaction rounds and target-model tokens after receiving tool access, while repeated-failure terminations increased from 1 to 40 for Plus and from 2 to 37 for Max. These results suggest that access to analytical tools alone does not guarantee better task performance or more efficient execution; effectively incorporating those tools into the execution workflow also matters. The supplementary experiment thus provides evidence about both the performance and execution costs associated with adding shared tools to CodeAct.
3. Semantic/VLM scoring, subject coverage, and consistency analyses
Only the first five source-ordered subjects per dataset are used, and parser agreement does not validate semantic/VLM judgments or quantify uncertainty from subjects and task templates.
Thank you for raising these valuable questions. We clarify our scoring procedure, subject selection, and existing consistency analyses below.
1. Semantic and VLM assessment follows explicit scoring instructions.
For semantic and visual assessment, we provide detailed scoring prompts specifying how the requested content and visual outputs should be evaluated, and use Qwen3.7-Max as a strong judge model. This assessment procedure is distinct from the extraction process examined in the parser audit.
2. The five-subject design balances recording diversity, analytical coverage, and evaluation cost.
BrainBench evaluates instruction-following and analytical execution by fixed LLM-based systems, rather than cross-subject predictive generalization in trained EEG decoders. Within each dataset, the common acquisition paradigm and recording setup allow the same analytical capabilities to be examined across multiple subjects under consistent acquisition conditions.
Selecting five subjects in a fixed source order provides reproducible, multi-subject coverage while keeping evaluation across datasets and tasks computationally manageable. Including every available subject would substantially increase repeated execution within the same dataset and task settings without necessarily expanding the analytical capabilities assessed. This design balances recording diversity with broader coverage of datasets and tasks.
3. Task-level uncertainty analysis.
Following your suggestion, we have quantified task-level uncertainty in the BrainAgent–CodeAct score differences, as detailed in our response to Q3.
Response to Reviewer rXm1[1/N]
We thank the reviewer for recognizing BrainBench’s broad coverage of EEG analysis beyond classification and the value of our analyses of failures and variability. We appreciate your careful comments on alignment between instructions and scoring, scientific validation, and the comparability and uncertainty of system evaluations. We address each concern below.
Main Comments
A1: We sincerely thank the reviewer for the careful reading and highlighting these points. We address these concerns in detail below.
1. FA-01: Task design and validation of numerical tolerances
For numerical tasks, we considered how reasonable differences in execution could affect the requested result. Where the instruction left implementation choices open, we examined alternative filter choices, the order of filtering and data selection, referencing conventions, and other relevant settings, provided that they remained compatible with the intended analysis. Variation across these reference calculations informed the numerical acceptance criteria. For stable, non-zero reference values, we typically used a tolerance of approximately 5% of the reference magnitude. Where acceptable implementation choices produced greater variation, we used 10% or a larger task-specific tolerance where justified. When unspecified processing choices produced materially different answers, we made the relevant requirements explicit in the instruction. For example, we specified the filter type or preprocessing order when these choices substantially affected the result. This approach combines task-specific tolerances with precise instructions to reduce penalties arising from unstated reference conventions. For FA-01, reference validation kept the required 0.5–40 Hz FIR preprocessing fixed, while examining variations in Welch PSD estimation, including 2- and 4-second windows, different overlap ratios, and Hann and Hamming windows. Based on this validation, we set the relative tolerance for FA-01 to 10% to accommodate reasonable implementation differences consistent with the requested analysis. The fixed Welch parameters in the reference code represent one reference implementation. Agents may use other scientifically justified pipelines that follow the instruction, with their outputs assessed against the task’s numerical acceptance criteria.
2. FA-30: Clarifying reporting requirements and improving instance selection
We sincerely thank the reviewer for identifying this task-design oversight and apologize for the unclear reporting requirement. Our intended output was the global mean PLV for each of the four bands, followed by identification and interpretation of the dominant synchronization band. We have made this requirement explicit:
For the cited instance, the four numerical PLVs remained within their acceptance tolerances across the tested implementations. However, our original instance selection did not adequately account for the sensitivity of a highest-band decision when the leading values are very close (many thanks for pointing this). The appropriate design is to select analysis windows with clearer separation and a stable ordering across the permitted implementations.
We therefore reviewed all 25 FA-30 instances and identified three with a relative gap between the two highest global mean PLV values: FA-30-01 (0.1108%), FA-30-03 (0.3265%), and FA-30-10 (0.0442%). We have replaced these three analysis windows with better-separated segments, verify their ranking stability across reasonable implementations, and update the corresponding instructions and reference targets. We are sincerely grateful to the reviewer for identifying this subtle but important aspect of task design that we had previously overlooked. Your careful observation has helped us further improve the rigor and reliability of our evaluation.