- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OmniBrain learns reusable EEG analysis guidance from successful and failed attempts on the same recording. A verification loop structures signal analysis, while Diagnostic Inference Graphs preserve measurement context for experience extraction. Evaluation spans six EEG classification tasks and Parkinson's cross-dataset transfer, without updating the agent's model weights.
Critical strengths
- The matched DIG-versus-direct extraction comparison uses the same records, attempt pairs, and library budgets (Section 5.3), providing useful evidence for the proposed representation.
- Evaluation across several tasks and the frozen-library transfer gains in Table 14 support the usefulness of reusable guidance beyond individual training recordings.
Critical weaknesses
- Several test sets are very small, including 8 OCD, 12 AD/FTD, and 13 schizophrenia recordings (Table 2). Three agent runs and single-seed foundation-model training do not quantify uncertainty across subjects or learned libraries. Subject-level paired intervals and run dispersion would strengthen the conclusions.
- Comparative superiority needs careful scope: pooled BIOT reaches 82.7% on Singh2020 versus OmniBrain's 78.85% (Table 13). The main frozen-backbone comparison also combines signal tools, an LLM, and multiple model predictions. These results support the workflow, but do not establish superiority over the strongest pooled alternative.
Questions for authors
- Do the three evaluation runs share one learned library, and can existing predictions provide subject-level paired uncertainty?
- How should the transfer claim be qualified in light of the stronger pooled baseline in Table 13?
5 comments
Q2 & W2. Transfer scope and comparative performance
[Q2] How should the transfer claim be qualified in light of the stronger pooled baseline in Table 13?
[W2] Comparative superiority needs careful scope: pooled BIOT reaches 82.7% on Singh2020 versus OmniBrain's 78.85% (Table 13). The main frozen-backbone comparison also combines signal tools, an LLM, and multiple model predictions. These results support the workflow, but do not establish superiority over the strongest pooled alternative.
Reply: A frozen experience library improves diagnosis across held-out targets without updating the agent’s weights. We show (1) this transfer gain, (2) the pooled-source ranking, and (3) the contributions of experience and FM access.
(1) The frozen Singh2021 experience library improves every transfer target with CEVL fixed. Mean BA rises from 64.52% to 76.63%, an absolute gain of 12.11% (Tab. 14). All four builds in Tab. S1 retain gains above 11% across the two case orders, without updating the agent’s weights.
(2) The pooled-source results give the cross-method ranking. Pooled end-to-end training uses 243 fitting recordings from five sources and 27 validation recordings, updating both backbone and head. In this setting, BIOT and BENDR have higher BA than OmniBrain on Singh2020.
Tab. R3. Pooled-source comparison on Singh2020, BA (%).
| Method | BA |
|---|---|
| CBraMod | 78.8 |
| LaBraM | 69.2 |
| BIOT | 82.7 |
| CSBrain | 75.0 |
| BENDR | 80.8 |
| OmniBrain | 78.85 |
(3) Experience contributes more BA than FM access in the component ablation. Tab. R1 keeps the LLM, signal tools, CEVL, and token budget fixed. FM-enabled methods receive the same four-FM predictions. Removing experience from the full agent reduces BA by an absolute 9.17–16.67%; removing FM access reduces it by 1.93–7.14%.
The FM-only LLM receives the four probability vectors, with EEG tools and experience disabled. Both conditions below use GPT-5.6-Sol at temperature 0.3.
Tab. R1. Selected controls, BA (%).
| Condition | PD | Albrecht2019 | Singh2020 |
|---|---|---|---|
| Agent with experience, without FMs | 70.33 | 71.43 | 76.92 |
| FM-only LLM: probabilities | 64.17 | 63.10 | 63.46 |
The no-FM agent exceeds the FM-only LLM in all three settings. Together with the experience on/off comparison, this shows the value of learned guidance beyond aggregating FM predictions.
In summary, frozen experience improves diagnosis on later datasets, and the component controls separate its contribution from FM access. The pooled-source results establish the comparative ranking under joint FM training.
We have added these analyses to the revised manuscript. Specifically, Appendix I.1 contains FM controls, I.2 experience library stability, I.5 transfer comparisons and PD averages, and I.6 confidence intervals.
Dear Reviewer 37jC,
Thank you for your detailed review and constructive questions. We would appreciate your feedback on the additional experiments and analyses in our response.
-
Frozen experience library transfer, component ablations, and pooled-source scope (W2, Q2). Transferring a frozen single-source experience library (Singh2021) raises mean balanced accuracy (BA) across all five held-out Parkinson's disease (PD) targets by 12.11% without weight updates, distinct from the five-source end-to-end weight-fine-tuning regime on held-out Singh2020 (69.2% to 82.7% across five foundation models (FMs), including BIOT at 82.7% and BENDR at 80.8%, vs. OmniBrain at 78.85%; Tab. R3, Appendix I.5). In a controlled 2×2 ablation (Tab. R1, Appendix I.1), removing experience lowers BA by 9.17% to 16.67% versus 1.93% to 7.14% when removing FM access, and the no-FM agent outperforms FM-only LLM aggregation in all three settings.
-
Shared experience library across evaluation runs and rebuild stability (W1, Q1). All three test runs per task share one frozen experience library with fixed instruction text and usage counts (Appendix I.2). Across four independent PD experience library builds on Singh2021 spanning original and shuffled training-case orders under matched budgets (Tab. S1), mean BA across the five held-out targets stays within 75.88% to 76.63% (a 0.75% span), consistently above the 64.52% no-experience baseline.
-
Subject-level paired confidence intervals (W1, Q1). Grouping each participant's recordings, subject-level paired 95% confidence intervals show a +14.13% BA gain ([6.02, 22.71]) over five-FM majority voting across nine datasets, alongside exact per-cohort paired intervals (Appendix I.6).
We hope these updates resolve your remaining questions and welcome any further feedback. Thank you again for your time and consideration.
Best regards,
Authors of #3883
Thank you for your helpful comments. We respond point by point below. Specifically, we clarify that inference runs share one frozen experience library per task and report paired participant uncertainty (Q1/W1). We show that transfer gains persist across four experience library builds (W1), and explain the benefit of transferring a frozen experience library, the pooled-source ranking, and the contributions of experience and FM access (Q2/W2).
Q1 & W1. Shared experience libraries and uncertainty
Reply: The inference runs share one frozen experience library per task. We clarify (1) what stays fixed during inference, (2) participant-level paired uncertainty, and (3) sensitivity to experience library construction.
(1) Instruction text and accumulated usage counts remain fixed during testing. Repeated inference uses the same completed experience library and training-case order. This evaluates the use of an existing experience library; independent construction is examined separately below.
(2) The intervals compare methods on the same recordings. Each participant’s recordings are kept together. FM training uses seed 42; cross-training-seed variation has not been measured.
(3) Rebuilding the experience library preserves the transfer gains across the tested case orders. Tab. S1 holds training cases, attempts per case, extraction settings, and the experience library budget fixed across four builds and two orders. The four mean BA values are 76.63%, 76.25%, 75.88%, and 76.50%; all exceed the no-experience mean of 64.52%.
In summary, the transfer gain persists across all four experience library builds and both case orders.