- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
EEGAgentBench evaluates LLM agents across six EEG tasks, comprising 1,072 instances with recording durations from two seconds to nearly 23 hours. It provides ten deterministic signal-processing tools and benchmarks 29 models, analyzing task performance, tool use, inference cost, and failures on long recordings.
Critical strengths
-
The benchmark covers diverse prediction formats, including classification, seizure localization, and full-recording sleep staging. A shared environment with deterministic tools provides a useful framework for comparing EEG agents across tasks and temporal scales.
-
The evaluation combines broad model coverage with informative metrics and trajectory analyses. Subject-averaged seizure F1, temporal Dice, and sleep-stage kappa reveal complementary failure modes that aggregate accuracy would miss.
Critical weaknesses
-
The long-horizon analysis does not adequately separate reasoning limitations from evidence acquisition and recording difficulty. Duration comparisons use different recordings, while case studies show tool-output limits followed by coarser measurements. Matched segments from the same recordings and a fixed, chunked feature-extraction baseline are needed to substantiate the central interpretation.
-
T3 confounds seizure status with recording modality: the healthy examples are scalp EEG, whereas the ictal examples are intracranial EEG. Performance could therefore partly reflect acquisition differences rather than epileptic-pattern interpretation. A comparison within the same recording modality is needed.
-
Difficulty filtering uses five models that subsequently appear in the evaluation, making the challenge distribution dependent on the screening panel. This substantially changes some tasks, removing 111 of 149 T1 questions. Rankings should be checked on the unfiltered pool or an independently selected subset.
-
Reproducibility and ranking stability remain insufficiently characterized. Models use different decoding configurations, but exact sampling and reasoning settings are not reported. Single-run evaluation without uncertainty estimates further limits comparisons, particularly for small tasks and extreme-duration groups.
Questions for authors
-
Does the duration-related performance decline persist on matched portions of the same recordings and when agents receive systematically extracted features through a fixed chunked workflow?
-
How sensitive are the overall rankings to difficulty filtering and to replacing T3 with a comparison within the same recording modality?