acceptodds
Under review as a conference paper at ICLR 2027

ERIA: An Instruction-Optimized Agent for Clinical EEG Reporting

Abstract

Clinical EEG interpretation is substantially more complex than classifying short signal segments; it must assess technical quality, characterize background and behavioral states across long recordings, identify interictal and ictal abnormalities, reconcile uncertain evidence, and synthesize these findings into a clinically meaningful report. We introduce ERIA (EEG Reporting and Instruction-Optimized Agent), a locally deployable EEG interpretation framework that enables a single 27B language-model controller to analyze whole-recording EEGs by orchestrating specialized parametric and signal-processing tools. ERIA combines specialized tools for detecting background abnormality, sleep–wake states, seizure-like activity, and interictal epileptiform discharges with deterministic spectral, amplitude, symmetry, montage, and signal-quality analyses, covering multiple clinically relevant domains reflected in EEG reporting guidelines. Rather than relying on EEG-specific controller fine-tuning, a proprietary frontier model, or multi-agent orchestration, ERIA keeps the controller weights, specialist tools, and safety constraints fixed and optimizes only its natural-language analysis policy with GEPA (Genetic-Pareto), a reflective prompt optimizer, using feedback from expert-written EEG reports. Compared with the raw policy, GEPA substantially improves physician-aligned whole-recording report quality, while the resulting policy transfers to external EEG benchmarks. We evaluate performance on CerebraGloss-Bench short-segment clinical interpretation and BrainBench Foundational Analysis, where ERIA performs competitively with the models evaluated in the original benchmarks. These results show that carefully selected specialist tools and language-level policy optimization can extend the clinical and general EEG capabilities of a single locally executable model while preserving local inference, traceable tool use, and a comparatively simple deployment architecture.

Primary area:
applications to neuroscience & cognitive science
Keywords:
clinical EEG, EEG interpretation, large language models, tool-using agents, clinical report generation, automatic prompt optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sw4gOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes an agentic framework for reviewing EEGs that is based around Qwen-27B as a backbone with an array of deep learning architecture tools used via a ReAct loop. The controller weights and tools remain fixed and the system’s prompt analysis policy is optimised by GEPA based on an LLM score against 100 reference reports.

Critical strengths

Clinical Scaffolding Alignment: Structuring the tool library and report around ACNS guidelines is clinically grounded and useful for downstream applications Traceable tool outputs: Unlike black-box EEG to text foundation models intermediate tool usage preserves the ability to audit signal findings

Critical weaknesses

Inadequate baselines: in Table 2 ERIA is only compared against itself, what about baselines such as CELM, EEGtoText or standard non agentic baselines. On Table 3 ERIA is substantially outperformed by CerebraGloss-3B on MCQ and ROUGE-1. On BrainBench the gain from GEPA is marginal. Lack of generalisation: This work is constructed around Qwen3.8-27B and not demonstrated to be effective on other backbones or model sizes leading to the question is this agentic framework useful or is it tailored to suit the specific design features of Qwen. Hallucination in prompt optimization: Section 5 mentions that GEPA optimised prompts ask for analyses which there is no tool to conduct demonstrating that prompt optimisation lacks grounding in the agent’s available tools. Absence of Ablations: No tool ablations despite 46.78 tool calls per report across 13 tools thus not detailing which learned models drive reporting score.

Questions for authors

Could the authors clarify the core technical contribution of ERIA over existing agentic EEG frameworks like EEGAgent? The architecture essentially wraps standard deterministic signal processing scripts and retrained classic classifiers in a standard ReAct loop, while applying an off-the-shelf prompt optimizer. Does the method generalise to other foundation model backbones? Does the method outperform current state-of-the-art for the same tasks? Which tools drive performance?

4 comments
AuthorsOpenReview ↗

Response to Reviewer sw4g part 1/2

We thank the reviewer for the detailed comments regarding ERIA’s technical contribution, baseline comparisons, tool design, and generalization. We have added several new analyses and experiments and are completing additional controlled studies to address these concerns.

1. Limited methodological contribution.

Thank you for raising this concern. Although our paper does not propose a fundamentally new agentic AI design for EEG interpretation, the uniqueness of our contribution lies in the clinically guided design of the toolset, computationally efficient and modular implementation, and a robust set of evaluations supporting ERIA’s deployment readiness. Specific contributions are highlighted below:

  • To our knowledge, ERIA is the first agentic system optimized to generate complete clinical EEG reports from long-duration recordings while combining an American Clinical Neurophysiology Society (ACNS)-aligned specialist-tool architecture. The proposed framework can also be extended to other medical data-interpretation tasks.
  • ERIA is fully local, supporting deployment in privacy-sensitive clinical environments. It is also modular, with plug-and-play components for the specialist toolset, LLM backbone, evaluator, reflector, and search algorithm. ERIA additionally has substantially lower computational overhead than existing agentic EEG frameworks such as EEGAgent.
  • We comprehensively evaluate the quality of ERIA-generated reports using multiple LLM evaluators and an ongoing human expert evaluation. Our evaluation also includes public benchmarks and transfer to unseen long-duration clinical EEG recordings.

2. Inadequate baselines:

To strengthen the evaluation beyond the original GPT-OSS judge, we rescored the complete validation and protected-test sets using two additional evaluators from different model families, Mistral-small:24b and Llama-3.3-70B. All three evaluators independently reproduce the Raw-to-GEPA improvement on the protected test set (table below). The improvement therefore does not depend on a single evaluator. We additionally report results for all seven clinical domains, where the direction of improvement is positive for all three evaluators.

SystemGPT-OSSMistral-small:24bLlama-3.3-70BMean
EEGAgent0.38380.49700.42540.4354
ERIA Raw0.50520.63870.58810.5773
ERIA GEPA0.60240.73140.70030.6780

We also added EEGAgent as an external agentic baseline on the same 100 protected test recordings (last row above), using the same Qwen3.8-27B controller as ERIA and providing the same structured clinical report format for final output. Across the 100 matched cases, Raw ERIA outperformed EEGAgent by +0.1419 in mean score across the three evaluators (95% paired-bootstrap CI:

+0.0894,+0.1949+0.0894, +0.1949

), while ERIA GEPA outperformed EEGAgent by +0.2426 (95% CI:

+0.1901,+0.2954+0.1901, +0.2954

). Both confidence intervals were estimated using 200,000 paired bootstrap resamples. The fact that Raw ERIA already substantially outperforms EEGAgent supports the value of ERIA’s clinically structured, ACNS-guided toolset and reporting framework independently of policy optimization; optimization further increases this separation.

3. EEG-to-text baselines.

We reviewed the publicly available EEG-to-text systems most relevant to clinical EEG reporting. However, we did not identify a released pretrained system that could be directly applied to our long-duration raw EEG recordings without substantial retraining, adaptation, or unavailable model components. CELM is the closest existing system, but its complete inference model is not currently available. We therefore added EEGAgent as the strongest directly executable EEG-analysis agent baseline.

4. Generalization across controller backbones.

We agree with the reviewer. We are currently repeating the protected-test experiment using Gemma 4 31B, a controller from a different foundation-model family at a comparable model scale. Importantly, the exact frozen GEPA-selected policy is transferred without reoptimization, while the specialist tools, fixed instructions, iteration budget, protected test set, and evaluation protocol remain unchanged. This directly tests whether the ERIA policy and framework transfer beyond the original Qwen controller. We will report the result once the ongoing run is complete.

5. Tool ablations.

Thank you for the comment. We are performing a controlled tool-perturbation experiment in which selected ERIA specialist tools are replaced using randomized tool substitutions while keeping the controller, policy, recordings, and evaluation protocol fixed. This experiment is computationally expensive, we will report the results when complete.

AuthorsOpenReview ↗

Response to Reviewer sw4g part 2/2

6. Contribution of individual specialist models.

ERIA does not execute a fixed sequence of specialist models. The controller selects tools adaptively according to the evidence required for each EEG recording. Consequently, the importance of an individual tool is highly case-dependent, and invocation frequency should not be interpreted as its causal contribution to the final clinical score.

Our full tool-use analysis shows that ERIA uses the specialist toolset broadly across the evaluation cohort, with significant variation across cases. We will report these usage statistics explicitly. The controlled tool experiment described above will provide a more direct assessment of whether the clinically aligned toolset contributes to performance.

ToolRaw calls/caseRaw SDGEPA calls/caseGEPA SD
spectral_analysis6.144.387.685.59
coarse_activity_screen5.143.055.723.07
fine_event_classifier4.241.745.682.72
background_abnormality3.562.153.043.29
amplitude_statistics3.463.264.584.62
sleep_wake_staging3.422.187.122.35
assess_signal_quality2.182.815.627.79
retrieve_eeg_knowledge1.920.592.340.51
hemispheric_symmetry1.420.702.301.24
ied_candidate_screen1.160.461.701.36
recording_summary1.000.001.000.00
inspect_montage0.080.270.000.00

7. Prompt/tool grounding & hallucinations.

Thank you for pointing this out. During optimization, the reflector is given the available tool descriptions as fixed context. However, upon further inspection, we found that the selected policy did contain one unsupported instruction encouraging inspection of sleep spindles, K-complexes, and vertex waves despite ERIA not having a dedicated detector for these morphologies. We will explicitly report this limitation and emphasize that optimized policies should be inspected for capability mismatches before deployment.

8. External-transfer robustness.

We agree that the reported BrainBench improvement is modest. BrainBench primarily evaluates targeted EEG-analysis questions that often do not require the same depth of iterative reasoning and evidence synthesis as generation of a complete long-duration clinical EEG report; accordingly, we would not expect policy optimization to produce a large Raw-to-GEPA gain on this benchmark. Importantly, the result shows that optimization for clinical report quality does not come at the expense of external transfer. We also note that even Raw ERIA is highly competitive with the frontier models reported on BrainBench, supporting the strength of the underlying ERIA architecture independently of GEPA. To quantify run-to-run variability rather than relying on a single realization, we are currently performing five BrainBench runs.

Regarding CerebraGloss outperforming ERIA on MCQ and ROUGE-1, the comparison is not architecture-matched. CerebraGloss-3B is fine-tuned for 10-second EEG image interpretation, whereas ERIA is designed for long recordings and relies on specialist tools, several of which require longer windows and are unavailable under the 10-second protocol. We therefore interpret CerebraGloss primarily as a transfer test: despite the task mismatch, GEPA does not materially degrade performance relative to Raw ERIA. We are also running five CerebraGloss repetitions to quantify variability.

AuthorsOpenReview ↗

Response to Reviewer sw4g part 3/3

Since our previous response, we have completed additional experiments. We report these results below and also clarify the remaining repeated external-benchmark evaluation.

Different LLM backbone.

As discussed above, we replaced the Qwen3.8-27B controller primarily used by ERIA with Gemma 4 31B, a similarly sized model from a different model family, while keeping everything else fixed. We evaluated both the Raw and GEPA-selected policies on the same 100-case protected test set.

ControllerPolicyGPT-OSSMistral-small:24bLlama-3.3-70BMean
Qwen3.8-27BRaw0.50520.63870.58810.5773
Qwen3.8-27BGEPA0.60240.73140.70030.6780
Gemma 4 31BRaw0.47930.57420.53660.5300
Gemma 4 31BGEPA0.56670.65260.63990.6197

The Raw-to-GEPA improvement transfers to Gemma and is reproduced independently by all three judges. On Gemma, GEPA improves the mean score by +0.0897, compared with +0.1007 on Qwen, corresponding to approximately 89% of the Qwen gain transferring without reoptimization. Moreover, Gemma with the Qwen-optimized GEPA policy achieves a mean score of 0.6197, exceeding Qwen with the Raw policy (0.5773). Thus, despite differences in the capabilities and reasoning behavior of the two controllers, a policy optimized using Qwen transfers effectively to a different backbone and improves that backbone beyond the original Qwen Raw baseline.

Tool ablations.

The tools contributions are shown in the per-domain score analysis, and the tool contributions are aggregated in a deterministic fashion as explained in the paper. Each domain mainly depends on a single learned tool. And we can't remove one of the tools because a complete domain would be missing. Also, ERIA's RAW and GEPA prompts are built in a way to coordinate these tools. Even if we assigned random probabilities to a single tool, ERIA would spend most of its reasoning effort trying to resolve the contradictions, wasting its compute budget. On the other hand, our existing experiments include ablations for the GEPA policy vs RAW, a different LLM backbone, the multi-LLM evaluators, and a human evaluation.

Repeated external-benchmark evaluation.

The five-run BrainBench and CerebraGloss evaluations remain to be completed and will be included in the final revision. We note that ERIA uses temperature-0 decoding for both benchmarks and that their evaluation pipelines are largely deterministic: BrainBench primarily uses structured deterministic metrics, while CerebraGloss uses deterministic scoring for ROUGE-1 and multiple-choice accuracy, with LLM-based evaluation used only for the conversational task. We will nevertheless report repeated-run variability for completeness.

Reviewer_sw4gOpenReview ↗

Response to rebuttal

Thank you for your response, adding additional LLM discriminators is not sufficient comparison with state of the art, nor is only one of the mentioned state of the art models. The generalisation ablations and tool ablations show promise, however this submission needs more effective benchmarking to meet the bar for ICLR publication. Thus I will maintain my score.

Reviewer_YwzeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ERIA, which uses a local 27B LLM as a single controller to call specialist EEG tools and generate clinical reports for entire recordings.

Critical strengths

S1. The system integrates complementary EEG tools for full-recording clinical reporting. Its locally deployable, single-controller architecture has a clear practical motivation and is technically feasible.

S2. Evaluation on a held-out test set and an external benchmark, together with tool-call trajectory analysis, provides preliminary evidence of changes in performance and analysis behavior.

Critical weaknesses

W1. The methodological contribution beyond EEGAgent and GEPA lies mainly in system integration and task adaptation. Controlled experiments provide limited support for the necessity and benefits of the key design choices.

W2. Validation of clinical report quality, usability, and end-to-end robustness remains limited.

Questions for authors

Q1. Could the authors add a comparison against an expert-designed prompt or a fixed workflow, using the same controller, tools, and analysis budget, to verify the gains from strategy optimization over simpler alternatives?

Q2. Has ERIA been used in real clinical workflows?

Q3. Do improvements in automated scores align with EEG experts’ judgments? Has agreement between evaluator scores and expert ratings been assessed?

If the system is already in real-world use, reporting data and analyses from its deployment would help clarify the value of this work. Given the many remaining unknowns, my current score is provisional.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Ywze

We thank the reviewer for the thoughtful comments regarding ERIA’s technical contribution and the need for stronger validation. We have added several analyses and experiments that directly address these concerns.

1. Limited methodological contribution.

We appreciate the reviewer’s point. Although our paper does not propose a fundamentally new agentic AI design for EEG interpretation, the uniqueness of our contribution lies in the clinically guided design of the toolset, computationally efficient and modular implementation, and a robust set of evaluations supporting ERIA’s deployment readiness. Specific contributions are highlighted below:

  • To our knowledge, ERIA is the first agentic system optimized to generate complete clinical EEG reports from long-duration recordings while combining an American Clinical Neurophysiology Society (ACNS)-aligned specialist-tool architecture. The proposed framework can also be extended to other medical data-interpretation tasks.
  • ERIA is fully local, supporting deployment in privacy-sensitive clinical environments. It is also modular, with plug-and-play components for the specialist toolset, LLM backbone, evaluator, reflector, and search algorithm. ERIA additionally has substantially lower computational overhead than existing agentic EEG frameworks such as EEGAgent.
  • We comprehensively evaluate the quality of ERIA-generated reports using multiple LLM evaluators and an ongoing human expert evaluation. Our evaluation also includes public benchmarks and transfer to unseen long-duration clinical EEG recordings.

2. Limited evaluations and comparisons against baselines.

To strengthen the evaluation beyond the original GPT-OSS judge, we rescored the complete validation and protected-test sets using two additional evaluators from different model families, Mistral-small:24b and Llama-3.3-70B. All three evaluators independently reproduce the Raw-to-GEPA improvement on the protected test set (table below). The improvement therefore does not depend on a single evaluator. We additionally report results for all seven clinical domains, where the direction of improvement is positive for all three evaluators.

SystemGPT-OSSMistral-small:24bLlama-3.3-70BMean
ERIA Raw0.50520.63870.58810.5773
ERIA GEPA0.60240.73140.70030.6780
EEGAgent0.38380.49700.42540.4354

We also added EEGAgent as an external agentic baseline on the same 100 protected test recordings (last row above), using the same Qwen3.8-27B controller as ERIA and providing the same structured clinical report format for final output. Across the 100 matched cases, Raw ERIA outperformed EEGAgent by +0.1419 in mean score across the three evaluators (95% paired-bootstrap CI:

+0.0894,+0.1949+0.0894, +0.1949

), while ERIA GEPA outperformed EEGAgent by +0.2426 (95% CI:

+0.1901,+0.2954+0.1901, +0.2954

). Both confidence intervals were estimated using 200,000 paired bootstrap resamples. The fact that Raw ERIA already substantially outperforms EEGAgent supports the value of ERIA’s clinically structured, ACNS-guided toolset and reporting framework independently of policy optimization; optimization further increases this separation.

3. Expert-designed prompt or fixed-workflow comparison.

Thank you for this comment. The Raw ERIA policy is already expert-guided. We highlight that our contribution is not a new prompt-search algorithm, and we do not claim that GEPA is uniquely necessary for improving the system. Comparisons with alternative optimization strategies, such as random search, would be valuable future work.

4. Real-world clinical workflow use.

ERIA has not yet been prospectively deployed. The present study is a retrospective evaluation using real clinical EEG recordings and paired physician reports. Prospective clinical workflow evaluation is an important next step. While the study is retrospective, our evaluations do show robust performance in multiple public benchmarks and in long duration clinical EEG recordings.

5. Agreement with EEG experts.

A blinded EEG-expert evaluation of 50 cases is currently in progress. We will report the expert ratings together with their agreement with the automated evaluators as soon as this evaluation is complete.

6. Additional evaluations on clinical report quality and end-to-end robustness.

Beyond the multi-evaluator analysis above, we added statistical support for the improvement on the 100-case protected test set. The GPT-OSS aggregate clinical score increased from 0.5052 to 0.6024 (+0.0972). A paired nonparametric bootstrap with 200,000 resamples yielded a 95% confidence interval of

+0.0483,+0.1470+0.0483, +0.1470

, and a two-sided paired sign-flip test with 1,000,000 permutations gave (p=0.000184). We are also performing an additional independent GEPA optimization run to assess the reproducibility of the optimization result.

Thanks for the additional analyses and clarifications. The new results help address some of my earlier concerns, but the following concerns remain:

  1. Stating that the raw policy is already expert-guided does not address the missing fixed-workflow comparison.

  2. The authors clarify that ERIA has not been prospectively deployed, and its value in real clinical workflows remains to be established.

  3. The current evidence remains limited regarding the added value of dynamic agent orchestration, clinical benefits, and operational reliability.

Given that the paper’s primary contribution lies in system design and engineering integration, I remain unconvinced that the current evidence supports acceptance at ICLR. I therefore maintain my original score.

We thank the reviewer for the follow-up. We have now completed the fixed-workflow comparison requested.

1. Fixed workflow versus adaptive orchestration.

We constructed a deterministic baseline using the same Qwen3.8-27B controller, ERIA specialist tools, GEPA-selected policy, protected 100-case test set, and a matched analysis budget. The fixed workflow used 48 predefined tool calls per report, obtained by rounding the mean GEPA tool-use frequencies. Tool identities, timestamps, and analysis windows were fixed in advance, and no tool output could trigger follow-up analyses. After the 48 calls, all tool outputs were provided to Qwen together with the same GEPA policy and original user prompt, with further tool access disabled.

SystemMistral-small:24b judge
Fixed workflow + GEPA0.6948
ERIA GEPA0.7314

Adaptive ERIA GEPA improves over the fixed workflow by +0.0366. This directly supports an additional benefit from adaptive evidence acquisition. Importantly, the fixed workflow is already a favorable control because its per-tool call allocation was derived from the average behavior of adaptive GEPA. If the workflow had been designed as a fixed pipeline from the outset, the appropriate number of calls to allocate to each tool would not have been known in advance. Thus, adaptive reasoning not only determines which analyses to perform for a given case, but also provides the empirical tool-use pattern used to construct this budget-matched fixed baseline.

2. Clinical validation and deployment.

We understand the reviewer's concern regarding this. ERIA has not yet been prospectively deployed. However, the system was evaluated retrospectively on 100 unseen clinical EEG recordings with paired physician reports. We also completed a blinded evaluation by an independent board-certified epileptologist on 30 randomly selected ERIA GEPA reports. The physician and GPT-OSS mean scores were 0.631 and 0.614, with strong case-level agreement (Pearson (r=0.905), Spearman (rho=0.806)); exact domain-level agreement was 71.0%, and 95.7% of numeric ratings differed by at most one level. These results provide additional evidence that ERIA’s report quality generalizes to unseen clinical cases and that the automated evaluator aligns with expert clinical judgment.

We sincerely appreciate the reviewer’s careful feedback and constructive suggestions, which have helped us strengthen both the evaluation and the presentation of the work.

Reviewer_iGb7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents ERIA, an agent that writes clinical EEG reports from whole recordings. A single locally run 27B language model acts as a controller in a reasoning and tool call loop. It never reads raw waveforms. It calls specialist tools and reads their structured outputs. The tools cover signal quality, montage, amplitude, spectra, and symmetry, plus small learned models for background abnormality, sleep and wake state, seizure-like activity, and interictal discharges, plus local text retrieval. Most learned tools are trained by the authors on public EEG corpora. The system prompt is split into a fixed part with safety and grounding rules and a mutable natural language policy. Only the policy is optimized, using GEPA, with feedback from an LLM judge that compares each generated report to a physician report across seven domains. The data are 200 private routine EEG studies with paired reports, split into 50 optimization, 50 validation, and 100 protected test cases. The main result is that the aggregate judge score on the protected test set rises from 0.5052 to 0.6024 after optimization, and tool call statistics show broader recording coverage. On external benchmarks, MCQ accuracy on CerebraGloss-Bench stays at 71.11% while the BrainBench Foundational Analysis score rises from 85.86 to 86.89. The contributions are the single local controller design, policy optimization driven by physician report feedback without weight updates, and evaluation against paired physician references in an agentic setting.

Critical strengths
  1. The paper gives clean evidence that optimizing only the natural language policy can materially improve a grounded clinical agent. The controller weights, the tools, and the safety rules stay fixed, and only the policy text changes, so the gain from 0.5052 to 0.6024 on a protected test set can be tied to the policy itself rather than to extra model capacity or new tools. The tool call analysis backs this up, since the optimized policy clearly gathers evidence differently, for example covering early, middle, and late parts of the recording in 98% of cases instead of 40%. This matters most for my assessment because it isolates a cheap and simple way to improve agents in settings where retraining is impractical, and the result is supported by two independent kinds of evidence, the score change and the behavior change.

  2. The work fills a real gap in agentic EEG analysis by using a single locally run 27B controller and evaluating whole recording reports against paired physician references on protected test cases. Prior agentic systems relied on much larger or proprietary controllers and did not evaluate report quality against matched physician reports. Hospitals often cannot send patient data to external providers, so a system that runs locally on one model, keeps tool use traceable, and needs the larger judge and reflector models only during offline optimization is a practical contribution rather than a demo. This is consequential because it makes the policy optimization result relevant to actual clinical deployment instead of only to a benchmark.

Critical weaknesses
  1. The entire clinical result rests on one LLM judge that is also the optimization target. GEPA receives feedback from the GPT-OSS-120b evaluator, and the same evaluator produces the final validation and test scores. The protected test set prevents overfitting to specific cases, but it does not prevent the policy from learning to write reports that this particular judge rewards, whether or not physicians would agree with them. The paper gives no human ratings on any subsample, no second judge as a cross check, no per domain scores, and no counts of NA domains, so there is no independent evidence that the gain from 0.5052 to 0.6024 reflects better clinical content rather than a better fit to the judge. The paper itself shows this risk is real, since it reports that the CerebraGloss-Bench judge score moved from 4.76 to 4.13 under the same protocol when the endpoint changed. This is consequential because the paper's central claim is about physician aligned report quality, and that claim currently depends entirely on the judgment of the model being optimized against.

  2. The optimization evidence comes from a single stochastic run with no statistical support. The GEPA reflector samples at temperature 1.0, only one run is reported, and there are no seeds, no reruns, no optimization curve, and no significance test on the test set gain of 0.0972 over 100 cases. There is also no baseline optimizer at equal budget, such as random search or a hand written policy, so even if the gain is real, the paper does not show that GEPA specifically is responsible rather than policy search in general. This is consequential because the method contribution is the claim that reflective policy optimization is what improves the agent, and a single un-replicated run with no comparison cannot carry that claim. If the gain shrinks or vanishes across seeds, the main result of the paper changes.

Questions for authors

Some questions on the existing contents reported in the paper:

  1. Is the 50, 50, 100 split done at the patient level, so that no patient appears in more than one partition? If the same patient can appear in both optimization and test, the protected test gain could be inflated.
  2. Can you report per domain scores for both policies on validation and test, together with the number of NA ratings per domain? The aggregate score hides where the gain comes from, and if ictal activity is NA for most routine studies, the stated weights do not reflect the objective that was actually optimized.
  3. In Table 2, the total of 46.78 calls over 9.34 iterations gives about 5.01 calls per iteration, not the reported 5.19. Which figure is incorrect, or how exactly was this ratio computed? Panel B also omits coarse activity screen calls, even though the coarse to fine strategy is central to the design. What were those counts?
  4. The raw policy had one invalid output on CerebraGloss-Bench that was excluded, so the raw scores average over 89 cases and the optimized scores over 90. Did the optimized run have any invalid outputs, and can you report the scores over matched denominators?
  5. Which BrainBench Foundational Analysis items are graded by an LLM, and what is the run to run variance of the 85.86 and 86.89 scores, given that extra tools were added to the system for this benchmark?

Something that I'd like to see additional experiments: 6. Can you validate the judge against physician ratings on a subsample, or replicate the main test result with a second judge model from a different model family? This is the most important addition, since the same judge currently supplies the optimization feedback and the final score, and the paper gives no independent evidence that higher judge scores mean better clinical reports. 7. Can you report at least one or two additional GEPA runs with different seeds, or at minimum the optimization curve and the number of accepted candidates, so that readers can tell whether the gain is typical rather than a single favorable run? 8. Can you include the raw and optimized policy texts, or representative diffs, together with the judge prompt and rubric, so that what was actually learned can be inspected directly?

6 comments
AuthorsOpenReview ↗

Response to Reviewer iGb7 part 1/2

We thank the reviewer for the careful assessment and for identifying several areas where additional validation and reporting would strengthen the manuscript. We have performed substantial additional analyses and are completing several further experiments.

1. Dependence on the GPT-OSS evaluator. We agree that improvement measured only by the same evaluator used during GEPA optimization would leave open the possibility of evaluator-specific overfitting. We therefore independently rescored both Raw and GEPA reports using two additional evaluators from different model families, Mistral-small:24b and Llama-3.3-70B. Importantly, both independent judges reproduce the Raw-to-GEPA improvement on both validation and protected test sets, and the improvement is positive across all seven clinical domains.

Validation set (n=50)

Clinical domainJudge: Prompt:GPT-OSS RAWGPT-OSS GEPAMistral RawMistral GEPALlama RawLlama GEPA
Aggregate clinical score0.50990.62770.63200.75690.59590.7229
Background0.34000.45000.62770.66000.52040.5700
Ictal activity0.80000.94000.86170.95000.84690.9700
Interictal epileptiform activity0.64000.69000.74470.79000.69390.7700
Overall classification0.61000.63000.55320.71000.44900.6500
State / sleep0.43000.61220.77780.85420.68370.8100
Focal / regional abnormality0.43750.54170.53660.66670.51140.6327
Impression0.21000.46000.44680.62000.40820.6200

Protected test set (n=100)

Clinical domainJudge: Prompt:GPT-OSS RawGPT-OSS GEPAMistral RawMistral GEPALlama RawLlama GEPA
Aggregate clinical score0.50520.60240.63870.73140.58810.7003
Background0.33000.41500.59900.63780.54120.6020
Ictal activity0.84000.93500.86980.96940.84540.9490
Interictal epileptiform activity0.63130.71720.71880.80100.64430.7602
Overall classification0.57000.59500.54690.66330.47420.6173
State / sleep0.37500.53000.75000.81960.68040.8010
Focal / regional abnormality0.47470.54590.61640.69150.50570.6302
Impression0.18500.37500.43750.56120.39180.5612

Thus, the improvement is not specific to GPT-OSS: the protected-test aggregate increases by +0.0972 with GPT-OSS, +0.0927 with Mistral, and +0.1122 with Llama.

2. GEPA stochasticity and optimization trajectory. We agree that a second optimization run would provide useful evidence regarding stochasticity. An additional independent GEPA run is currently being performed. These runs are computationally expensive because every candidate requires repeated whole-recording agent execution and evaluation, but we will report the additional run once complete. We have also reconstructed the complete optimization trajectory for the reported run. GEPA admitted 12 candidate policies including the seed; the best validation score progressed from 0.5099 for C0 to 0.5610 for C1, 0.5632 for C4, and 0.6277 for the selected C9. We will add the candidate-lineage figure and expose the complete Raw policy, selected GEPA policy, and evaluator rubric in the appendix.

3. Simpler prompt-optimization alternatives. Thanks for the comment. Our contribution is not a new prompt-search algorithm, and we do not intend to claim that GEPA is uniquely necessary. The Raw ERIA policy is already expert-guided and serves as the non-optimized expert baseline using the identical controller and toolset. We will clarify this scope. Equal-budget comparisons with alternative optimization strategies such as random search are valuable future work.

4. Patient-level separation. The 200 recordings come from 200 unique patients; no patient appears in more than one train, validation, or test partition. We will make this explicit.

5. Domain reporting, NA handling, and statistical support. We will also report the complete domain-level results above and NA counts. NA ratings were rare: 5 of 700 validation-domain evaluations and 5 of 1,400 test-domain evaluations. On the 100 paired protected-test cases, the aggregate clinical score increased from 0.5052 with the Raw policy to 0.6024 with GEPA, corresponding to an absolute improvement of +0.0972. To quantify uncertainty while preserving the within-case pairing, we performed a paired nonparametric bootstrap with 200,000 resamples, yielding a 95% confidence interval of [+0.0483, +0.1470]. We additionally performed a two-sided paired sign-flip permutation test with 1,000,000 permutations, which confirmed that the observed improvement was statistically significant (p=0.000184p=0.000184).

AuthorsOpenReview ↗

Response to Reviewer iGb7 part 2/2

6. Calls per iteration and tool accounting. In Table 2, the two values were obtained using different aggregation procedures: (46.78/9.34=5.01) is the ratio of the mean number of tool calls to the mean number of controller iterations, whereas 5.19 is the mean of the 50 per-case calls-per-iteration ratios. Both are mathematically valid but summarize the data differently. To remove ambiguity, we will use a single definition consistently in the revision and state the calculation explicitly.

We also agree that coarse_activity_screen should be shown because it is central to ERIA's coarse-to-fine workflow. Panel B was originally intended to highlight selected tools with the largest or most clinically interpretable changes rather than provide an exhaustive tool inventory. We will add coarse_activity_screen to the main table (5.14 to 5.72 calls/case) and provide the complete tool-use table in the appendix. inspect_preprocessing was not called by either Raw or GEPA.

ToolRaw totalRaw calls/caseGEPA totalGEPA calls/case
spectral_analysis3076.143847.68
coarse_activity_screen2575.142865.72
fine_event_classifier2124.242845.68
background_abnormality1783.561523.04
amplitude_statistics1733.462294.58
sleep_wake_staging1713.423567.12
assess_signal_quality1092.182815.62
retrieve_eeg_knowledge961.921172.34
hemispheric_symmetry711.421152.30
ied_candidate_screen581.16851.70
recording_summary501.00501.00
inspect_montage40.0800.00
Total tool calls168633.72233946.78

7. CerebraGloss robustness. The Raw CerebraGloss run produced one empty output on the description-generation task evaluated with ROUGE-1, whereas GEPA produced valid description outputs for all 90 cases. Following the released CerebraGloss evaluation protocol, the invalid Raw output was excluded rather than assigned a score of zero, yielding Raw (n=89) and GEPA (n=90). For transparency, we will additionally report both alternative matched comparisons: assigning the missing Raw output a score of zero gives Raw ROUGE-1 = 30.36 versus GEPA = 30.17, while restricting both systems to the same 89 valid cases gives Raw = 30.70 versus GEPA = 30.03.. We also agree that the variation observed across evaluator endpoints indicates that the absolute judge score should not be overinterpreted. Our intended conclusion from CerebraGloss is therefore conservative: optimization did not materially impair external transfer. To quantify run-to-run variability directly, we are currently performing five repeated CerebraGloss evaluations.

8. BrainBench variance. We agree that the observed BrainBench gain is modest and should not be presented as a large external-benchmark improvement. Its main value is that the optimized policy preserves, rather than sacrifices, performance under external transfer. We are currently performing five BrainBench runs to quantify run-to-run variance and will report the mean and variability rather than relying on a single run.

9. Human EEG-expert validation. We agree that agreement with an independent EEG expert is the most important remaining validation. A blinded expert evaluation of 50 cases is currently in progress, and we will add these results and their agreement with the automated evaluators as soon as the evaluation is complete.

10. Raw and optimized policy texts, or representative diffs, together with the judge prompt and rubric. We agree that exposing the policy and evaluator prompts improves transparency. We did not include these lengthy prompt artifacts in the original manuscript because they were intended to be provided with the released code. However, in the final PDF, we will add them directly to the appendix.

We will update the remaining results once complete.

Reviewer_iGb7OpenReview ↗

Reply to rebuttal

Thanks for adding the Mistral and Llama judge tables, the paired bootstrap CI, and the full tool-call counts. Those help clear up part of my earlier questions.

That said, there are still major gaps and a few new concerns from the numbers you shared, so I don't think the paper is ready yet:

  1. Too many core checks are still "in progress" (and prompts are still missing): Most of the key validations across the three reviews—the 50-case clinician evaluation, a second GEPA run, the Gemma backbone check, tool ablations, and the 5-run variance on BrainBench/CerebraGloss—are still unfinished, and my question on which BrainBench items use an LLM judge wasn't answered. Also, pasting the raw policy, the GEPA policy, and the judge rubric takes zero compute, yet we still can't see what prompt GEPA actually learned or verify there isn't other unsupported instructions besides the sleep-spindle one.

  2. What the per-domain scores show: On the protected test set under GPT-OSS, Overall classification (Normal vs Abnormal) barely moves at all (0.570 to 0.595, not far above 0.50), while Background (0.415) and Impression (0.375) both stay below 0.50 ("partially correct"). Meanwhile, Ictal activity (20% weight) has almost no NAs and goes from 0.840 to 0.935 on 100 routine EEGs where seizures are almost always absent—so just predicting "no seizures" gives near-full credit on a big chunk of the score. Without sensitivity/specificity or counts on true positive abnormal cases, we can't tell if GEPA actually detects events better or just got more conservative.

  3. Baselines and transfer: Once CerebraGloss denominators are matched, ROUGE-1 slightly drops after GEPA (30.70 to 30.03 on n=89) and MCQ is flat (71.11%). For the new EEGAgent baseline, shrinking its backbone from 235B to 27B and grading it on sleep and spike domains where EEGAgent lacks ERIA's tools isn't an apples-to-apples test, and we still miss the simpler fixed-workflow baseline (running ERIA's tools on fixed early/mid/late windows without a 47-call ReAct loop) that Reviewer Ywze also asked about.

I appreciate the ongoing work, and I'll keep my score at 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.