- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes an agentic framework for reviewing EEGs that is based around Qwen-27B as a backbone with an array of deep learning architecture tools used via a ReAct loop. The controller weights and tools remain fixed and the system’s prompt analysis policy is optimised by GEPA based on an LLM score against 100 reference reports.
Critical strengths
Clinical Scaffolding Alignment: Structuring the tool library and report around ACNS guidelines is clinically grounded and useful for downstream applications Traceable tool outputs: Unlike black-box EEG to text foundation models intermediate tool usage preserves the ability to audit signal findings
Critical weaknesses
Inadequate baselines: in Table 2 ERIA is only compared against itself, what about baselines such as CELM, EEGtoText or standard non agentic baselines. On Table 3 ERIA is substantially outperformed by CerebraGloss-3B on MCQ and ROUGE-1. On BrainBench the gain from GEPA is marginal. Lack of generalisation: This work is constructed around Qwen3.8-27B and not demonstrated to be effective on other backbones or model sizes leading to the question is this agentic framework useful or is it tailored to suit the specific design features of Qwen. Hallucination in prompt optimization: Section 5 mentions that GEPA optimised prompts ask for analyses which there is no tool to conduct demonstrating that prompt optimisation lacks grounding in the agent’s available tools. Absence of Ablations: No tool ablations despite 46.78 tool calls per report across 13 tools thus not detailing which learned models drive reporting score.
Questions for authors
Could the authors clarify the core technical contribution of ERIA over existing agentic EEG frameworks like EEGAgent? The architecture essentially wraps standard deterministic signal processing scripts and retrained classic classifiers in a standard ReAct loop, while applying an off-the-shelf prompt optimizer. Does the method generalise to other foundation model backbones? Does the method outperform current state-of-the-art for the same tasks? Which tools drive performance?
4 comments
Response to Reviewer sw4g part 2/2
6. Contribution of individual specialist models.
ERIA does not execute a fixed sequence of specialist models. The controller selects tools adaptively according to the evidence required for each EEG recording. Consequently, the importance of an individual tool is highly case-dependent, and invocation frequency should not be interpreted as its causal contribution to the final clinical score.
Our full tool-use analysis shows that ERIA uses the specialist toolset broadly across the evaluation cohort, with significant variation across cases. We will report these usage statistics explicitly. The controlled tool experiment described above will provide a more direct assessment of whether the clinically aligned toolset contributes to performance.
| Tool | Raw calls/case | Raw SD | GEPA calls/case | GEPA SD |
|---|---|---|---|---|
| spectral_analysis | 6.14 | 4.38 | 7.68 | 5.59 |
| coarse_activity_screen | 5.14 | 3.05 | 5.72 | 3.07 |
| fine_event_classifier | 4.24 | 1.74 | 5.68 | 2.72 |
| background_abnormality | 3.56 | 2.15 | 3.04 | 3.29 |
| amplitude_statistics | 3.46 | 3.26 | 4.58 | 4.62 |
| sleep_wake_staging | 3.42 | 2.18 | 7.12 | 2.35 |
| assess_signal_quality | 2.18 | 2.81 | 5.62 | 7.79 |
| retrieve_eeg_knowledge | 1.92 | 0.59 | 2.34 | 0.51 |
| hemispheric_symmetry | 1.42 | 0.70 | 2.30 | 1.24 |
| ied_candidate_screen | 1.16 | 0.46 | 1.70 | 1.36 |
| recording_summary | 1.00 | 0.00 | 1.00 | 0.00 |
| inspect_montage | 0.08 | 0.27 | 0.00 | 0.00 |
7. Prompt/tool grounding & hallucinations.
Thank you for pointing this out. During optimization, the reflector is given the available tool descriptions as fixed context. However, upon further inspection, we found that the selected policy did contain one unsupported instruction encouraging inspection of sleep spindles, K-complexes, and vertex waves despite ERIA not having a dedicated detector for these morphologies. We will explicitly report this limitation and emphasize that optimized policies should be inspected for capability mismatches before deployment.
8. External-transfer robustness.
We agree that the reported BrainBench improvement is modest. BrainBench primarily evaluates targeted EEG-analysis questions that often do not require the same depth of iterative reasoning and evidence synthesis as generation of a complete long-duration clinical EEG report; accordingly, we would not expect policy optimization to produce a large Raw-to-GEPA gain on this benchmark. Importantly, the result shows that optimization for clinical report quality does not come at the expense of external transfer. We also note that even Raw ERIA is highly competitive with the frontier models reported on BrainBench, supporting the strength of the underlying ERIA architecture independently of GEPA. To quantify run-to-run variability rather than relying on a single realization, we are currently performing five BrainBench runs.
Regarding CerebraGloss outperforming ERIA on MCQ and ROUGE-1, the comparison is not architecture-matched. CerebraGloss-3B is fine-tuned for 10-second EEG image interpretation, whereas ERIA is designed for long recordings and relies on specialist tools, several of which require longer windows and are unavailable under the 10-second protocol. We therefore interpret CerebraGloss primarily as a transfer test: despite the task mismatch, GEPA does not materially degrade performance relative to Raw ERIA. We are also running five CerebraGloss repetitions to quantify variability.
Response to Reviewer sw4g part 3/3
Since our previous response, we have completed additional experiments. We report these results below and also clarify the remaining repeated external-benchmark evaluation.
Different LLM backbone.
As discussed above, we replaced the Qwen3.8-27B controller primarily used by ERIA with Gemma 4 31B, a similarly sized model from a different model family, while keeping everything else fixed. We evaluated both the Raw and GEPA-selected policies on the same 100-case protected test set.
| Controller | Policy | GPT-OSS | Mistral-small:24b | Llama-3.3-70B | Mean |
|---|---|---|---|---|---|
| Qwen3.8-27B | Raw | 0.5052 | 0.6387 | 0.5881 | 0.5773 |
| Qwen3.8-27B | GEPA | 0.6024 | 0.7314 | 0.7003 | 0.6780 |
| Gemma 4 31B | Raw | 0.4793 | 0.5742 | 0.5366 | 0.5300 |
| Gemma 4 31B | GEPA | 0.5667 | 0.6526 | 0.6399 | 0.6197 |
The Raw-to-GEPA improvement transfers to Gemma and is reproduced independently by all three judges. On Gemma, GEPA improves the mean score by +0.0897, compared with +0.1007 on Qwen, corresponding to approximately 89% of the Qwen gain transferring without reoptimization. Moreover, Gemma with the Qwen-optimized GEPA policy achieves a mean score of 0.6197, exceeding Qwen with the Raw policy (0.5773). Thus, despite differences in the capabilities and reasoning behavior of the two controllers, a policy optimized using Qwen transfers effectively to a different backbone and improves that backbone beyond the original Qwen Raw baseline.
Tool ablations.
The tools contributions are shown in the per-domain score analysis, and the tool contributions are aggregated in a deterministic fashion as explained in the paper. Each domain mainly depends on a single learned tool. And we can't remove one of the tools because a complete domain would be missing. Also, ERIA's RAW and GEPA prompts are built in a way to coordinate these tools. Even if we assigned random probabilities to a single tool, ERIA would spend most of its reasoning effort trying to resolve the contradictions, wasting its compute budget. On the other hand, our existing experiments include ablations for the GEPA policy vs RAW, a different LLM backbone, the multi-LLM evaluators, and a human evaluation.
Repeated external-benchmark evaluation.
The five-run BrainBench and CerebraGloss evaluations remain to be completed and will be included in the final revision. We note that ERIA uses temperature-0 decoding for both benchmarks and that their evaluation pipelines are largely deterministic: BrainBench primarily uses structured deterministic metrics, while CerebraGloss uses deterministic scoring for ROUGE-1 and multiple-choice accuracy, with LLM-based evaluation used only for the conversational task. We will nevertheless report repeated-run variability for completeness.
Response to rebuttal
Thank you for your response, adding additional LLM discriminators is not sufficient comparison with state of the art, nor is only one of the mentioned state of the art models. The generalisation ablations and tool ablations show promise, however this submission needs more effective benchmarking to meet the bar for ICLR publication. Thus I will maintain my score.
Response to Reviewer sw4g part 1/2
We thank the reviewer for the detailed comments regarding ERIA’s technical contribution, baseline comparisons, tool design, and generalization. We have added several new analyses and experiments and are completing additional controlled studies to address these concerns.
1. Limited methodological contribution.
Thank you for raising this concern. Although our paper does not propose a fundamentally new agentic AI design for EEG interpretation, the uniqueness of our contribution lies in the clinically guided design of the toolset, computationally efficient and modular implementation, and a robust set of evaluations supporting ERIA’s deployment readiness. Specific contributions are highlighted below:
2. Inadequate baselines:
To strengthen the evaluation beyond the original GPT-OSS judge, we rescored the complete validation and protected-test sets using two additional evaluators from different model families, Mistral-small:24b and Llama-3.3-70B. All three evaluators independently reproduce the Raw-to-GEPA improvement on the protected test set (table below). The improvement therefore does not depend on a single evaluator. We additionally report results for all seven clinical domains, where the direction of improvement is positive for all three evaluators.
We also added EEGAgent as an external agentic baseline on the same 100 protected test recordings (last row above), using the same Qwen3.8-27B controller as ERIA and providing the same structured clinical report format for final output. Across the 100 matched cases, Raw ERIA outperformed EEGAgent by +0.1419 in mean score across the three evaluators (95% paired-bootstrap CI:
+0.0894,+0.1949), while ERIA GEPA outperformed EEGAgent by +0.2426 (95% CI:
+0.1901,+0.2954). Both confidence intervals were estimated using 200,000 paired bootstrap resamples. The fact that Raw ERIA already substantially outperforms EEGAgent supports the value of ERIA’s clinically structured, ACNS-guided toolset and reporting framework independently of policy optimization; optimization further increases this separation.
3. EEG-to-text baselines.
We reviewed the publicly available EEG-to-text systems most relevant to clinical EEG reporting. However, we did not identify a released pretrained system that could be directly applied to our long-duration raw EEG recordings without substantial retraining, adaptation, or unavailable model components. CELM is the closest existing system, but its complete inference model is not currently available. We therefore added EEGAgent as the strongest directly executable EEG-analysis agent baseline.
4. Generalization across controller backbones.
We agree with the reviewer. We are currently repeating the protected-test experiment using Gemma 4 31B, a controller from a different foundation-model family at a comparable model scale. Importantly, the exact frozen GEPA-selected policy is transferred without reoptimization, while the specialist tools, fixed instructions, iteration budget, protected test set, and evaluation protocol remain unchanged. This directly tests whether the ERIA policy and framework transfer beyond the original Qwen controller. We will report the result once the ongoing run is complete.
5. Tool ablations.
Thank you for the comment. We are performing a controlled tool-perturbation experiment in which selected ERIA specialist tools are replaced using randomized tool substitutions while keeping the controller, policy, recordings, and evaluation protocol fixed. This experiment is computationally expensive, we will report the results when complete.