acceptodds
Under review as a conference paper at ICLR 2027

OmniBrain: An Experience-Driven Agent for Evidence-Grounded Clinical EEG Diagnosis

Abstract

Electroencephalography (EEG) diagnosis requires selecting measurements and interpreting ambiguous findings in clinical context. Foundation models provide diagnostic predictions, but these outputs do not specify which findings warrant further checks or how to assess them in context. Learning reusable diagnostic guidance is challenging because case labels evaluate final diagnoses, leaving intermediate analysis decisions unlabeled. We introduce OmniBrain, an EEG agent that learns interpretation and verification guidance from diagnostic feedback without updating model weights. Within each case, a Clinical Evidence Verification Loop (CEVL) organizes hypothesis-driven reasoning: stating expectations before analysis, interpreting measurements in context, and pursuing follow-up checks. The resulting diagnostic records are represented as Diagnostic Inference Graphs (DIGs) that preserve measurement context and evidence dependencies. Comparing successful and failed DIGs from the same recording holds the input fixed, exposing interpretation differences and unmatched follow-up checks. An LLM converts these contrasts into reusable guidance for subsequent cases. At inference, the learned experience guides signal analysis, after which the agent consults foundation-model predictions to form its final diagnosis. Against five frozen-backbone foundation models and their majority vote, OmniBrain achieves the highest task-level balanced accuracy on five of six EEG tasks. DIG-based extraction outperforms direct extraction on both evaluated tasks, and experience learned from one Parkinson's disease dataset transfers across five held-out datasets.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG, clinical diagnosis, LLM agent, experience learning, EEG Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_37jCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OmniBrain learns reusable EEG analysis guidance from successful and failed attempts on the same recording. A verification loop structures signal analysis, while Diagnostic Inference Graphs preserve measurement context for experience extraction. Evaluation spans six EEG classification tasks and Parkinson's cross-dataset transfer, without updating the agent's model weights.

Critical strengths
  • The matched DIG-versus-direct extraction comparison uses the same records, attempt pairs, and library budgets (Section 5.3), providing useful evidence for the proposed representation.
  • Evaluation across several tasks and the frozen-library transfer gains in Table 14 support the usefulness of reusable guidance beyond individual training recordings.
Critical weaknesses
  • Several test sets are very small, including 8 OCD, 12 AD/FTD, and 13 schizophrenia recordings (Table 2). Three agent runs and single-seed foundation-model training do not quantify uncertainty across subjects or learned libraries. Subject-level paired intervals and run dispersion would strengthen the conclusions.
  • Comparative superiority needs careful scope: pooled BIOT reaches 82.7% on Singh2020 versus OmniBrain's 78.85% (Table 13). The main frozen-backbone comparison also combines signal tools, an LLM, and multiple model predictions. These results support the workflow, but do not establish superiority over the strongest pooled alternative.
Questions for authors
  1. Do the three evaluation runs share one learned library, and can existing predictions provide subject-level paired uncertainty?
  2. How should the transfer claim be qualified in light of the stronger pooled baseline in Table 13?
5 comments

Thank you for your helpful comments. We respond point by point below. Specifically, we clarify that inference runs share one frozen experience library per task and report paired participant uncertainty (Q1/W1). We show that transfer gains persist across four experience library builds (W1), and explain the benefit of transferring a frozen experience library, the pooled-source ranking, and the contributions of experience and FM access (Q2/W2).

Q1 & W1. Shared experience libraries and uncertainty

[Q1] Do the three evaluation runs share one learned library, and can existing predictions provide subject-level paired uncertainty?

[W1] Several test sets are very small, including 8 OCD, 12 AD/FTD, and 13 schizophrenia recordings (Table 2). Three agent runs and single-seed foundation-model training do not quantify uncertainty across subjects or learned libraries. Subject-level paired intervals and run dispersion would strengthen the conclusions.

Reply: The inference runs share one frozen experience library per task. We clarify (1) what stays fixed during inference, (2) participant-level paired uncertainty, and (3) sensitivity to experience library construction.

(1) Instruction text and accumulated usage counts remain fixed during testing. Repeated inference uses the same completed experience library and training-case order. This evaluates the use of an existing experience library; independent construction is examined separately below.

(2) The intervals compare methods on the same recordings. Each participant’s recordings are kept together. FM training uses seed 42; cross-training-seed variation has not been measured.

(3) Rebuilding the experience library preserves the transfer gains across the tested case orders. Tab. S1 holds training cases, attempts per case, extraction settings, and the experience library budget fixed across four builds and two orders. The four mean BA values are 76.63%, 76.25%, 75.88%, and 76.50%; all exceed the no-experience mean of 64.52%.

In summary, the transfer gain persists across all four experience library builds and both case orders.

Q2 & W2. Transfer scope and comparative performance

[Q2] How should the transfer claim be qualified in light of the stronger pooled baseline in Table 13?

[W2] Comparative superiority needs careful scope: pooled BIOT reaches 82.7% on Singh2020 versus OmniBrain's 78.85% (Table 13). The main frozen-backbone comparison also combines signal tools, an LLM, and multiple model predictions. These results support the workflow, but do not establish superiority over the strongest pooled alternative.

Reply: A frozen experience library improves diagnosis across held-out targets without updating the agent’s weights. We show (1) this transfer gain, (2) the pooled-source ranking, and (3) the contributions of experience and FM access.

(1) The frozen Singh2021 experience library improves every transfer target with CEVL fixed. Mean BA rises from 64.52% to 76.63%, an absolute gain of 12.11% (Tab. 14). All four builds in Tab. S1 retain gains above 11% across the two case orders, without updating the agent’s weights.

(2) The pooled-source results give the cross-method ranking. Pooled end-to-end training uses 243 fitting recordings from five sources and 27 validation recordings, updating both backbone and head. In this setting, BIOT and BENDR have higher BA than OmniBrain on Singh2020.

Tab. R3. Pooled-source comparison on Singh2020, BA (%).

MethodBA
CBraMod78.8
LaBraM69.2
BIOT82.7
CSBrain75.0
BENDR80.8
OmniBrain78.85

(3) Experience contributes more BA than FM access in the component ablation. Tab. R1 keeps the LLM, signal tools, CEVL, and token budget fixed. FM-enabled methods receive the same four-FM predictions. Removing experience from the full agent reduces BA by an absolute 9.17–16.67%; removing FM access reduces it by 1.93–7.14%.

The FM-only LLM receives the four probability vectors, with EEG tools and experience disabled. Both conditions below use GPT-5.6-Sol at temperature 0.3.

Tab. R1. Selected controls, BA (%).

ConditionPDAlbrecht2019Singh2020
Agent with experience, without FMs70.3371.4376.92
FM-only LLM: probabilities64.1763.1063.46

The no-FM agent exceeds the FM-only LLM in all three settings. Together with the experience on/off comparison, this shows the value of learned guidance beyond aggregating FM predictions.

In summary, frozen experience improves diagnosis on later datasets, and the component controls separate its contribution from FM access. The pooled-source results establish the comparative ranking under joint FM training.

We have added these analyses to the revised manuscript. Specifically, Appendix I.1 contains FM controls, I.2 experience library stability, I.5 transfer comparisons and PD averages, and I.6 confidence intervals.

Dear Reviewer 37jC,

Thank you for your detailed review and constructive questions. We would appreciate your feedback on the additional experiments and analyses in our response.

  • Frozen experience library transfer, component ablations, and pooled-source scope (W2, Q2). Transferring a frozen single-source experience library (Singh2021) raises mean balanced accuracy (BA) across all five held-out Parkinson's disease (PD) targets by 12.11% without weight updates, distinct from the five-source end-to-end weight-fine-tuning regime on held-out Singh2020 (69.2% to 82.7% across five foundation models (FMs), including BIOT at 82.7% and BENDR at 80.8%, vs. OmniBrain at 78.85%; Tab. R3, Appendix I.5). In a controlled 2×2 ablation (Tab. R1, Appendix I.1), removing experience lowers BA by 9.17% to 16.67% versus 1.93% to 7.14% when removing FM access, and the no-FM agent outperforms FM-only LLM aggregation in all three settings.

  • Shared experience library across evaluation runs and rebuild stability (W1, Q1). All three test runs per task share one frozen experience library with fixed instruction text and usage counts (Appendix I.2). Across four independent PD experience library builds on Singh2021 spanning original and shuffled training-case orders under matched budgets (Tab. S1), mean BA across the five held-out targets stays within 75.88% to 76.63% (a 0.75% span), consistently above the 64.52% no-experience baseline.

  • Subject-level paired confidence intervals (W1, Q1). Grouping each participant's recordings, subject-level paired 95% confidence intervals show a +14.13% BA gain ([6.02, 22.71]) over five-FM majority voting across nine datasets, alongside exact per-cohort paired intervals (Appendix I.6).

We hope these updates resolve your remaining questions and welcome any further feedback. Thank you again for your time and consideration.

Best regards,

Authors of #3883

Reviewer_SVatOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OmniBrain is an EEG diagnostic agent that learns reusable guidance from successful and failed analyses of the same recording. A Clinical Evidence Verification Loop structures tool-based reasoning, while Diagnostic Inference Graphs organize evidence for experience extraction. The resulting library guides subsequent analyses, with EEG foundation-model predictions consulted before the final classification.

Critical strengths
  1. The framework connects executable EEG measurements with explicit hypotheses, interpretations, and follow-up checks, making the diagnostic process more inspectable than a final prediction alone.

  2. The DIG ablation matches source records, extraction settings, and library budgets, showing substantial improvements over direct extraction. A frozen experience library also improves performance across five transfer datasets, supporting the usefulness of reusable guidance.

Critical weaknesses
  1. Comparing successful and failed attempts does not establish which intermediate decisions were correct or useful. Every reported skill receives credit from the final diagnosis, while existing instructions remain unchanged. Consequently, incorrect or incidental reasoning may be retained without adequate validation.

  2. OmniBrain receives predictions from four foundation models at inference, but the experiments do not isolate their contribution through a prediction-only LLM aggregator or an agent without FM access. This leaves unclear how much improvement comes from signal-based reasoning versus combining existing predictions.

  3. The highlighted transfer comparison gives OmniBrain experience from five source datasets but compares it with separately trained single-source models. Under pooled-source training, BIOT and BENDR achieve 82.7% and 80.8% BA, respectively, exceeding OmniBrain’s 78.85%.

  4. Several test sets are very small, including 8 OCD, 12 AD/FTD, and 13 schizophrenia recordings. Three inference runs without confidence intervals do not establish the reliability of the gains or their sensitivity to experience-library construction

Questions for authors
  1. How are unreliable learned instructions identified or corrected, and how sensitive are results to training-case order and independent library-construction runs?

  2. Can the authors provide prediction-only and no-FM agent controls, together with subject-level paired confidence intervals for the principal comparisons?

6 comments

Thank you for your constructive comments. We respond point by point below. Specifically, we show that DIG-based experience improves held-out diagnosis and retains gains across four experience library builds (Q1/W1/W4). We also explain instruction checks and usage feedback (Q1/W1), isolate experience from FM aggregation (Q2/W2), clarify source coverage in transfer (W3), and provide the requested participant-level uncertainty (Q2/W4).

Q1 & W1. Instruction reliability and experience library construction

[Q1] How are unreliable learned instructions identified or corrected, and how sensitive are results to training-case order and independent library-construction runs?

[W1] Comparing successful and failed attempts does not establish which intermediate decisions were correct or useful. Every reported skill receives credit from the final diagnosis, while existing instructions remain unchanged. Consequently, incorrect or incidental reasoning may be retained without adequate validation.

Reply: We check candidate instructions against their source evidence and track the outcomes of later uses. We explain (1) how instructions are checked and tracked, then show (2) their value under matched extraction and (3) the effect of rebuilding the experience library.

(1) Candidate instructions are checked against the measurements and reasoning that produced them. DIG comparison identifies interpretation differences and missing follow-up checks between successful and failed attempts on the same recording. An LLM formulates each instruction and its applicability condition, then checks it against the linked evidence. If the source interpretation remains unresolved, the instruction specifies the evidence needed to assess it.

Later feedback updates usage counts; existing instruction text stays fixed. During training, every instruction reported as used receives a correct-use or incorrect-use count according to the final diagnosis. These are case-outcome records that the agent sees when considering instructions for later cases. Both instruction text and accumulated counts are frozen at test time.

(2) DIG-based extraction produces more useful guidance from the same diagnostic attempts. The comparison holds training recordings, successful/failed attempt pairs, the LLM, consolidation, and experience library budgets fixed. Both experience libraries are evaluated with the same CEVL procedure.

Tab. R2. Experience extraction, BA (%).

Experience settingPDAlbrecht2019
No experience65.3361.90
Direct textual extraction62.0854.76
DIG-based extraction74.5078.57

DIG-based guidance improves over both direct textual extraction and the no-experience agent. Direct extraction falls below the no-experience reference on both tasks. Preserving measurement context and evidence dependencies therefore improves the guidance obtained from these attempt pairs.

(3) The transfer gain persists across four experience library builds and two case orders. We hold the Singh2021 training cases, attempts per case, extraction settings, and experience library budget fixed. Each completed experience library is frozen before evaluation on the five transfer targets.

Tab. S1. Four experience library builds, mean BA (%).

Case orderExperience libraryMean BA (%)
OriginalSubmitted experience library76.63
OriginalIndependent rebuild76.25
ShuffledIndependent build 175.88
ShuffledIndependent build 276.50

The four builds give mean BA between 75.88% and 76.63%, a range of 0.75%. All remain above the no-experience mean of 64.52%.

In summary, instructions remain linked to their source evidence, and later outcomes inform their use through counts. The matched extraction experiment and independent builds show that the resulting experience improves diagnoses beyond the cases used to construct it.

Q2 & W2. FM-only and no-FM controls

[Q2] Can the authors provide prediction-only and no-FM agent controls, together with subject-level paired confidence intervals for the principal comparisons?

[W2] OmniBrain receives predictions from four foundation models at inference, but the experiments do not isolate their contribution through a prediction-only LLM aggregator or an agent without FM access. This leaves unclear how much improvement comes from signal-based reasoning versus combining existing predictions.

Reply: Experience improves diagnosis even without FM predictions. We show (1) its contribution with FM access enabled or disabled and (2) the comparison with FM-only aggregation. Paired intervals are given under W4 below.

(1) Experience improves the agent under both FM-access conditions. We keep CEVL, signal tools, the LLM, and the token budget fixed. Every FM-enabled method uses the same checkpoints and recording-level outputs from CBraMod, LaBraM, BIOT, and CSBrain. Agent and LLM conditions use GPT-5.6-Sol at temperature 0.3.

Tab. S3. FM controls, BA (%).

ConditionPDAlbrecht2019Singh2020
Full OmniBrain74.5078.5778.85
Agent without FMs70.3371.4376.92
Agent without experience65.3361.9065.38
Agent without FMs or experience58.5054.7657.69
FM-only LLM: probabilities64.1763.1063.46
Probability averaging69.8847.6278.85
Logistic-regression stacking65.6361.9075.00

Without FMs, adding experience gives absolute BA gains of 11.83–19.23% across the three settings. Experience also improves all three settings when FM access is enabled.

The experience library guides analysis before any FM predictions are read. Interpretation rules help assess measured patterns; verification rules guide the next check, such as checking channel quality before interpreting power asymmetry. CEVL applies this guidance through targeted measurements and follow-up verification, giving the final decision EEG evidence beyond the FM outputs.

(2) The no-FM agent exceeds the FM-only LLM in every setting. The LLM control receives only the four probability vectors, with EEG tools and experience disabled. Logistic-regression stacking is fitted on source-validation probabilities, with Singh2020 held out. The full agent leads these controls on PD and Albrecht2019 and ties probability averaging on Singh2020. The additional controls include label-only LLM aggregation and four-FM voting.

W3. Source coverage in the transfer comparison

The highlighted transfer comparison gives OmniBrain experience from five source datasets but compares it with separately trained single-source models. Under pooled-source training, BIOT and BENDR achieve 82.7% and 80.8% BA, respectively, exceeding OmniBrain's 78.85%.

Reply: The frozen experience library improves diagnosis on all five transfer targets. We distinguish this gain from the pooled-source performance comparison.

The pooled comparison matches the five-source coverage. OmniBrain’s system-level setting uses experience from five datasets, while each single-source checkpoint uses one. Under pooled training, BIOT and BENDR score higher than OmniBrain on Singh2020.

Reusing Singh2021 experience improves diagnosis on every target. With CEVL fixed, the frozen experience library yields an absolute mean BA gain of 12.11% across the five targets (Tab. 14), without updating the agent’s weights.

W4 & Q2. Uncertainty across participants and experience libraries

Several test sets are very small, including 8 OCD, 12 AD/FTD, and 13 schizophrenia recordings. Three inference runs without confidence intervals do not quantify uncertainty across subjects or learned libraries. Subject-level paired intervals and run dispersion would strengthen the conclusions

Reply: We quantify participant uncertainty and test sensitivity to experience library construction. We report (1) paired participant-level intervals and (2) results from independent experience library builds.

(1) The intervals compare methods on the same recordings. Each participant’s recordings are kept together.

(2) The four experience library builds give similar results. Tab. S1 shows their mean BA values, which range from 75.88% to 76.63% across two case orders with the training cases, extraction settings, and experience library budget fixed. Each exceeds the no-experience result.

We have added these analyses to the revised manuscript. Specifically, Appendix I.1 contains FM controls, I.2 instruction updates and experience library stability, I.5 transfer comparisons, and I.6 confidence intervals.

Dear Reviewer SVat,

Thank you for your detailed review and constructive questions. We would appreciate your feedback on the additional experiments and analyses in our response.

  • Isolating learned experience from foundation-model aggregation (W2, Q2). Evaluated against FM-only LLM aggregators, probability averaging, majority voting, and logistic-regression stacking on the same four foundation-model (FM) predictions (Tab. S3, Appendix I.1), learned experience without FM outputs raises balanced accuracy (BA) by 11.83% to 19.23% across Parkinson's disease (PD), schizophrenia (Albrecht2019), and held-out PD transfer (Singh2020), outperforming FM-only LLM aggregation in every setting. Full OmniBrain leads on PD and schizophrenia and ties probability averaging on Singh2020.

  • Validating learned instructions and experience library stability (W1, W4, Q1). Before entering the experience library, candidate instructions from paired Diagnostic Inference Graphs (DIGs) on the same recording are checked against linked measurements, while case-outcome counts update during training and stay frozen at test time (Appendix I.2). Under matched budgets, DIG extraction beats direct text extraction by 12.42% to 23.81% BA (Tab. R2), and four independent PD experience library builds across original and shuffled case orders stay within a 0.75% span (75.88% to 76.63% mean transfer BA; Tab. S1).

  • Frozen experience library transfer, pooled-source protocols, and subject-level confidence intervals (W3, W4, Q2). Reusing a frozen experience library from a single PD source (Singh2021) raises mean BA across all five held-out PD targets by 12.11% without weight updates, whereas Appendix I.5 separately reports the joint five-source end-to-end weight-fine-tuning regime on held-out Singh2020 (69.2% to 82.7% across five FMs, including BIOT at 82.7% and BENDR at 80.8%, vs. OmniBrain at 78.85%). Subject-level paired 95% confidence intervals (grouping each participant's recordings) show a +14.13% BA gain ([6.02, 22.71]) over five-FM majority voting across nine datasets, alongside exact per-cohort paired intervals (Appendix I.6).

We would appreciate your reconsideration of the rating in light of these results. We welcome any further discussion. Thank you again for your time and consideration.

Best regards,

Authors of #3883

Reviewer_CM2dOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents OMNIBRAIN, an EEG diagnostic agent framework designed to learn reusable diagnostic guidance from case-level feedback without updating model parameters. The core workflow centers on a Clinical Evidence Verification Loop (CEVL) that structures hypothesis formation, expectation-driven measurement, and follow-up verification. Diagnostic attempts are represented as Diagnostic Inference Graphs (DIGs), where contrasts between successful and failed attempts on the same recording are used to induce reusable interpretation and verification rules. At test inference, the agent applies this learned experience and consults predictions from four frozen EEG foundation models (FMs) before synthesizing a final diagnosis. The framework is evaluated across six EEG classification benchmarks spanning Parkinson's disease, schizophrenia, AD/FTD, OCD, mTBI, and depression.

Critical strengths
  • The idea of contrasting successful and failed diagnostic trajectories on the exact same recording to extract guidance is sensible and well-motivated for EEG, as holding the input signal fixed effectively controls for severe inter-subject variability.
  • Structuring the diagnostic reasoning around executable signal processing tools with provenance tracking and artifact checks is a practical, grounded design that is much more credible than having language models hallucinate interpretations from raw values.
  • The ablation comparing DIG-based extraction against direct textual extraction with matched attempt pairs and token budgets is a solid experiment that clearly highlights the benefit of preserving structured evidence dependencies.
  • Cross-dataset transfer of diagnostic experience on Parkinson's disease shows promising gains (12.11% balanced accuracy on average across five target datasets over the no-experience baseline).
  • The paper produces traceable reports, and the case study on Mumtaz2016 recording 55 clearly illustrates how paired measurement contrasts can translate into explicit verification rules.
Critical weaknesses
  • I am not convinced with the choices and implementation of the baselines. All FM baselines are run with a single fixed random seed (seed 42) without model-specific tuning. Different architectures favor different training schedules, and a single rigid protocol can hide their true potential. Some reported baseline numbers are questionable: for example, end-to-end CBraMod achieves only 32.1% balanced accuracy on binary schizophrenia and 28.3% on 3-class AD/FTD, while frozen CSBrain and majority voting obtain negative Cohen's kappa values on AD/FTD and OCD. Do these numbers make sense and reflect the true task difficulty?
  • The evaluation sets for several tasks are far too small to support confident claims without error bars (e.g., 8 test recordings in Gruendler2009, 12 in ds004504, and 13 in Albrecht2019). Given these tiny sets, provding confidence internals will be necessary.
  • The gains across tasks are quite mixed when inspected closely. On mTBI, OMNIBRAIN's gain in Table 1 is merely +0.29% balanced accuracy over the simple majority vote baseline, while actually achieving lower Cohen's kappa (58.21 vs. 62.37) and lower F1 (79.24 vs. 82.65). On individual datasets, OMNIBRAIN underperforms the baselines on ASEEG (73.86% vs. 78.41% for CBraMod and MV), Mumtaz2016 (82.50% vs. 100.0% for LaBraM, BIOT, and MV), and clinical OCD (67.86% vs. 70.24% for LaBraM, CSBrain, and MV).
  • Here FMs is a confounding factor but is not ablated properly. At test time, the agent reads predictions from four FM before assigning a label. OMNIBRAIN is therefore evaluated as a meta-ensemble, but it is primarily compared against individual frozen FMs and a simple unweighted majority vote. The authors do not provide critical control configurations: OMNIBRAIN without some FM access, an LLM prompt that only reads FM predictions, or simple classifier stacking. Notably, in Table 15, CEVL-only without experience reaches 65.33% balanced accuracy on Parkinson's disease, which remains below the standalone frozen CBraMod model (69.71%).
  • The cross-dataset transfer result on Singh2020 appears overstated. In the main text, OMNIBRAIN's 78.85% is compared against single-source checkpoints (averaging 72.7% for BENDR). However, in the fair pooled-source setting reported in Table 13, fine-tuned baselines outperform OMNIBRAIN: BIOT achieves 82.7% and BENDR achieves 80.8%. Additionally, the paper frequently shifts protocols between frozen-backbone and end-to-end models across tables without consistent justification, making diffifult to follow.
  • There are several unexplained anomalies in the results. In Table 14, the Singh2021 source-experience performance on Singh2021 (held-out source) is reported as 66.25%, yet the mean over the five target datasets is reported as 76.63%, which does not seem to align with the values in Table 15 where OMNIBRAIN is 74.50% and CEVL-only is 65.33%. In Figure 6b and Table 15, the tool-using agent without CEVL scores 21.43% balanced accuracy on schizophrenia, which is drastically below chance and should be explained. In Mumtaz2016, the agent overrides FMs that achieved 100% accuracy, lowering the final task score to 82.50%.
Questions for authors
  • Could you provide an ablation of OMNIBRAIN operating purely on signal tools without querying FM predictions at test time, as well as an LLM baseline that performs stacking directly on FM predictions?
  • Why is the comparison against pooled FMs (Table 13), where BIOT (82.7%) and BENDR (80.8%) surpass OMNIBRAIN (78.85%), relegated to the appendix while the main text highlights only the comparison against single-source models?
  • Could you explain why the tool-using agent without CEVL dropped to 21.43% balanced accuracy on schizophrenia? Was there a systematic class-prediction bias here?
  • How do you account for the fact that in Mumtaz2016, multiple foundation models achieved 100% balanced accuracy, but the agent's interpretation loop overrode them to produce 82.50%?
  • Could you clarify the recording-level validation split on datasets with multiple sessions per subject (such as Cavanagh2018a and Brown2020), and confirm whether subjects were strictly isolated during checkpoint selection?
6 comments

Thank you for your detailed comments. We respond point by point below. Specifically, we show that experience improves diagnosis without FM predictions and DIG outperforms matched textual extraction (Q1/W4), with transfer gains from a frozen experience library evaluated alongside pooled-source models (Q2/W5). We clarify baseline adaptation, subject separation, and the PD averages (W1/Q5/W6). We also explain the cited diagnostic outcomes and dataset differences (Q3/Q4/W3) and provide the requested participant-level uncertainty (W2).

Q1 & W4. Is the gain explained by FM aggregation?

[Q1] Could you provide an ablation of OMNIBRAIN operating purely on signal tools without querying FM predictions at test time, as well as an LLM baseline that performs stacking directly on FM predictions?

[W4] Here FMs is a confounding factor but is not ablated properly. At test time, the agent reads predictions from four FM before assigning a label. OMNIBRAIN is therefore evaluated as a meta-ensemble, but it is primarily compared against individual frozen FMs and a simple unweighted majority vote. The authors do not provide critical control configurations: OMNIBRAIN without some FM access, an LLM prompt that only reads FM predictions, or simple classifier stacking. Notably, in Table 15, CEVL-only without experience reaches 65.33% balanced accuracy on Parkinson's disease, which remains below the standalone frozen CBraMod model (69.71%).

Reply: Learned experience improves diagnosis even without FM predictions. We show that (1) experience improves the agent with and without FM access, and (2) DIG-based extraction outperforms matched direct textual extraction.

(1) Experience adds diagnostic value beyond FM aggregation. Every FM-enabled method uses the same checkpoints and recording-level outputs from CBraMod, LaBraM, BIOT, and CSBrain. Agent conditions use CEVL, the same signal tools and token budget, and GPT-5.6-Sol at temperature 0.3. The manuscript’s five-FM vote additionally includes BENDR.

The FM-only LLM receives only the four FM labels or probability vectors. It uses the same LLM and temperature, with EEG tools and experience disabled. Stacking is fitted on source-validation probabilities, with Singh2020 held out. The shared Brown2020 and Cavanagh2018a heads use subject-disjoint fitting and validation.

Tab. R1. Balanced accuracy (%).

ConditionFM accessExperiencePDAlbrecht2019Singh2020
Full OmniBrainYesYes74.5078.5778.85
Agent without FMsNoYes70.3371.4376.92
Agent without experienceYesNo65.3361.9065.38
Agent without eitherNoNo58.5054.7657.69
FM-only LLM: labelsYesNo61.7963.1061.54
FM-only LLM: probabilitiesYesNo64.1763.1063.46
Probability averagingYesNo69.8847.6278.85
Majority vote (4 FMs)YesNo66.8848.8171.15
Logistic-regression stackingYesNo65.6361.9075.00

Without FM access, experience improves BA by 11.83% on PD, 16.67% on Albrecht2019, and 19.23% on Singh2020. With FMs enabled, the gains are 9.17%, 16.67%, and 13.47%. Removing FM access from the full agent causes smaller decreases in all three settings.

The cited 65.33% PD score is the agent without experience. With experience and no FM access, the agent reaches 70.33%, compared with CBraMod’s 69.71%. The full agent leads the controls on PD and Albrecht2019 and ties probability averaging on Singh2020. On Singh2020, probability averaging is higher than the no-FM agent.

(2) DIG-based extraction improves on matched direct-text extraction. The DIG and direct-text conditions use the same training recordings, successful/failed attempt pairs, LLM, consolidation, and experience library budgets. Both experience libraries are evaluated with CEVL.

Tab. R2. Experience extraction, BA (%).

Experience settingPDAlbrecht2019
No experience65.3361.90
Direct textual extraction62.0854.76
DIG-based extraction74.5078.57

DIG-based extraction gains 12.42% on PD and 23.81% on Albrecht2019 over direct textual extraction. It also improves on the no-experience reference. Direct extraction falls below that reference on both tasks. In the joint ablation in Tab. 15, combining CEVL and experience gives the highest BA among the tested conditions.

In summary, case-level experience improves diagnosis even without FM predictions. Preserving measurement context and evidence dependencies during extraction makes that experience more useful than the matched direct-text alternative.

Q2 & W5. Pooled-source transfer and training protocols

Why is the comparison against pooled FMs (Table 13), where BIOT (82.7%) and BENDR (80.8%) surpass OMNIBRAIN (78.85%), relegated to the appendix while the main text highlights only the comparison against single-source models? ... the paper frequently shifts protocols between frozen-backbone and end-to-end models across tables without consistent justification ...

Reply: Frozen experience improves diagnosis across held-out datasets without updating the agent’s weights. We show (1) the pooled-source comparison and (2) the gain from transferred experience.

(1) Pooled training matches the five-source coverage. OmniBrain uses experience from five datasets; each single-source FM uses one. Pooled FMs use 243 fitting and 27 validation recordings, with Singh2020 held out. Tab. 1 trains classification heads with frozen backbones; Tabs. 13 and 17 update both backbone and head. In the pooled setting, BIOT and BENDR have higher BA than OmniBrain on Singh2020.

Tab. R3. Pooled-source comparison on Singh2020, BA (%).

MethodBA
CBraMod78.8
LaBraM69.2
BIOT82.7
CSBrain75.0
BENDR80.8
OmniBrain78.85

(2) The frozen experience library improves all five transfer targets. With CEVL fixed, adding the Singh2021 experience library raises mean BA from 64.52% to 76.63% in Tab. 14. This 12.11% absolute gain measures the benefit of transferred guidance without updating the agent’s weights.

W6. Alignment of the PD averages

... the mean over the five target datasets is reported as 76.63%, which does not seem to align with the values in Table 15 where OMNIBRAIN is 74.50% and CEVL-only is 65.33%.

Reply: The averages cover different datasets. We clarify both scopes below.

Tab. 15 includes Singh2021; the transfer mean replaces it with Singh2020. Singh2021 alone scores 66.25%. Together with Cavanagh2018a/b, Singh2018, and Brown2020, it gives 74.50%; the replacement gives 76.63%. The no-experience means are 65.33% and 64.52%. Q3–Q4 cover the other anomalies.

W1. Baseline settings and low CBraMod scores

All FM baselines are run with a single fixed random seed (seed 42) without model-specific tuning. ... end-to-end CBraMod achieves only 32.1% balanced accuracy on binary schizophrenia and 28.3% on 3-class AD/FTD, while frozen CSBrain and majority voting obtain negative Cohen's kappa values on AD/FTD and OCD.

Reply: Each FM baseline is individually adapted. CBraMod’s official code yields consistent results under the same training configuration. We clarify the model-specific settings, official-code check, and κ values below.

(1) The models use their own input pipelines, heads, and schedules. FM training uses seed 42; variation across training seeds has not been measured.

(2) We verified CBraMod using its official code. The official implementation and our implementation produce consistent results when the training settings are matched.

(3) CSBrain and MV have negative κ on AD/FTD and positive task-average κ on OCD. Negative κ denotes agreement below that expected from the true-label and predicted-label marginals (Tab. 1).

Q5. Subject separation during checkpoint selection

Could you clarify the recording-level validation split ... and confirm whether subjects were strictly isolated during checkpoint selection?

Reply: Fitting and validation share subjects on two datasets. Test subjects are fully isolated.

On Brown2020 and Cavanagh2018a, different recordings from one participant can enter both fitting and validation. Test subjects are disjoint from both across all 13 cohorts. Shared Cavanagh2018a/b participants have consistent train/test assignments.

We compared both validation splits using frozen-backbone CBraMod at seed 42. The training pool, test set, hyperparameters, and selection metric were fixed.

Tab. R6. Validation-split comparison.

DatasetValidation splitValidation recordingsSelected epochTest BA (%)
Brown2020Recording-level12175.00
Brown2020Subject-disjoint8175.00
Cavanagh2018aRecording-level11153.13
Cavanagh2018aSubject-disjoint6153.13

Subject-disjoint validation preserves checkpoint selection and test predictions on both datasets. The shared FM panel in Tab. R1 uses subject-disjoint fitting and validation for Brown2020 and Cavanagh2018a.

Q3. Why the tool-only schizophrenia score is below chance

Could you explain why the tool-using agent without CEVL dropped to 21.43% balanced accuracy on schizophrenia? Was there a systematic class-prediction bias here?

Reply: The LLM agent evaluated via Claude Code frequently produced false positives in our motivating experiment. Recent studies also report false-positive tendencies [1] and diagnoses reached without sufficient evidence [2]. These findings motivated CEVL’s expectation-driven measurements and follow-up verification.

The ablation varies CEVL and experience while keeping the signal tools and analysis budget fixed. With tool use alone, the agent misclassifies controls as patients and also misses patient cases, explaining the below-chance result. CEVL requires expectations before analysis, interpretation of measurements in context, and follow-up checks for unresolved hypotheses. Performance improves with CEVL and improves further when experience is added.

[1] Evaluating large language models for assessment of psychosis risk. npj Digital Medicine, 2026.

[2] Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine, 2024.

Q4. Mumtaz2016 errors despite correct FM references

How do you account for the fact that in Mumtaz2016, multiple foundation models achieved 100% balanced accuracy, but the agent's interpretation loop overrode them to produce 82.50%?

Reply: OmniBrain weighs EEG evidence, artifact checks, experience, and FM predictions. It may therefore follow the EEG evidence even when all FMs predict otherwise.

(1) Mumtaz2016 errors involved regional beta. Beta surviving artifact screening led to two false positives; cleaner beta below threshold and downweighted artifact-sensitive features led to one false negative.

(2) EEG evidence can correct unanimous FM errors. Cavanagh2018a sub33_001 was correctly labelled no_parkinsons based on preserved alpha without persistent slowing, despite four parkinsons inputs. Trace inputs differ from seed-42 references.

W3. Mixed gains across tasks

On mTBI, OMNIBRAIN's gain in Table 1 is merely +0.29% ... while actually achieving lower Cohen's kappa ... and lower F1 ... On individual datasets, OMNIBRAIN underperforms the baselines on ASEEG ... Mumtaz2016 ... and clinical OCD ...

Reply: OmniBrain improves depressive-symptom classification and external mTBI diagnosis over the FM references. We show (1) these gains and (2) the within-source comparisons raised in the review.

(1) The gains extend across distinct EEG settings. On ds003478, BA is 81.32%, above the strongest FM’s 66.38%. On external mTBI, BA is 64.17%, versus 48.84% for MV and 59.36% for CSBrain, the strongest FM. This set contains 50 recordings from 20 participants excluded from source training, validation, and testing.

(2) The within-source results depend on the dataset and metric. The FM references have higher BA on ASEEG, Mumtaz2016, and clinical OCD. Within-source mTBI gains 0.29% BA over MV, with lower κ and weighted F1. Q4 explains the Mumtaz2016 decisions through the agent’s evidence integration.

The matched controls show gains from experience with and without FM access (Tab. R1). They identify the contribution of learned guidance to this diagnostic workflow.

W2. Uncertainty on small test sets

The evaluation sets for several tasks are far too small to support confident claims without error bars (e.g., 8 test recordings in Gruendler2009, 12 in ds004504, and 13 in Albrecht2019). Given these tiny sets, providing confidence intervals will be necessary.

Reply: We provide participant-level confidence intervals. They use paired predictions on the same recordings, with repeated and shared recordings from each participant kept together.

We have added these analyses to the revised manuscript. Specifically, Appendix I.1 contains FM controls, I.3 baseline and validation checks, I.4 diagnostic cases, I.5 transfer comparisons and PD averages, and I.6 confidence intervals.

Comments on the paper

1 comment

Dear Area Chairs and Reviewers,

Thank you for taking the time to review our work and for your thoughtful feedback.

(1) Overview of our work

Clinical diagnosis often requires deciding which findings to trust and what to check next. We developed OmniBrain to learn from earlier EEG cases which measurements to make and when a finding needs a closer look, while keeping the model weights fixed.

  • Learning from the same recording. OmniBrain compares successful and failed analyses to identify interpretations and follow-up checks that can guide later cases.
  • Preserving the evidence behind a decision. Diagnostic Inference Graphs link measurements to interpretations and subsequent checks. Given the same attempts, extracting guidance from these graphs yielded absolute gains in balanced accuracy of 12.42% on Parkinson’s disease and 23.81% on one schizophrenia dataset over direct textual extraction.

(2) Key strengths recognized by the reviewers

  • Traceable reasoning. CM2d and SVat highlighted that readers can inspect the EEG measurements and artifact checks behind a diagnosis.
  • Controlled comparison. CM2d noted that fixing the recording controls for differences between participants. All three reviewers highlighted the matched comparison of graph-based and textual extraction.
  • Transfer to new cases. CM2d, SVat, and 37jC recognized the gains from reusing a frozen experience library on held-out datasets.

(3) Responses to the reviewers’ concerns

  • Contribution beyond foundation-model predictions. With foundation-model access disabled, adding experience improved balanced accuracy in all three settings. In each setting, the agent using EEG tools and experience also outperformed the LLM given only foundation-model predictions. (CM2d Q1/W4, SVat Q2/W2, 37jC W2)
  • Instruction reliability and stability. We check candidate instructions against their source measurements and track the outcomes of later uses. The three inference runs per task shared one frozen experience library. Across four builds and two case orders, mean transfer balanced accuracy ranged from 75.88% to 76.63%. (SVat Q1/W1/W4, 37jC Q1/W1)
  • Transfer across datasets. We clarified the pooled-source comparison and isolated the contribution of transferred experience by holding the verification procedure fixed. Reusing experience from one Parkinson’s disease cohort raised mean balanced accuracy from 64.52% to 76.63% across five held-out datasets. (CM2d Q2/W5, SVat W3, 37jC Q2/W2)
  • Baseline evaluation and uncertainty. The foundation models used model-specific input pipelines, classifier heads, and training schedules. For frozen-backbone CBraMod, subject-disjoint validation selected the same checkpoint and yielded identical test predictions on both multi-session datasets examined. We also reported paired participant-level confidence intervals for the three small cohorts and for the mean balanced accuracy difference across nine datasets. (CM2d W1/W2/Q5, SVat W4/Q2, 37jC W1/Q1)

(4) Reviewer feedback

We responded to CM2d and 37jC on baseline comparisons, transfer, and uncertainty. In their follow-up, SVat wrote that the rebuttal “has addressed most of my concerns.”

We have included the additional results and clarifications in Appendix I of the revised manuscript. Thank you for considering our revised work.

Sincerely,
Authors of Submission #3883

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.