acceptodds
Under review as a conference paper at ICLR 2027

PRISM: Patient-specific Reasoning and Information Seeking for Multimodal Psychiatric Diagnosis

Abstract

Multimodal reasoning under partial observability requires acquiring evidence whose decision value justifies its cost. However, predefined prediction pipelines provide limited support for adapting evidence sources and analysis operations to evolving information needs. We formulate psychiatric diagnosis from electronic health records (EHR) and electroencephalography (EEG) as a patient-specific reasoning and information-seeking problem and propose PRISM, a large language model (LLM) agent framework for joint source and analysis selection. First, we construct PsyNexus, a longitudinal multimodal cohort of 2,051 patients with major depressive disorder, bipolar disorder, or schizophrenia, comprising 2,799 encounter-linked EEG recordings and 58,514 EHR encounters. The encounter linkage supports interpreting neural observations within each patient's clinical context. Second, we develop PRISM-Base, an evidence-grounded agent that maintains a patient-level diagnostic state and integrates clinical facts, explicit EEG measurements, and optional learned representations through source-linked tools. Third, we introduce Progressive Evidence Seeking and Reasoning (PESR), a reinforcement learning alignment strategy for resource-efficient evidence acquisition and analysis. PESR evaluates diagnostic refinement with a frozen assessor and progressively introduces evidence grounding, information value, and cost sensitivity, allowing informative acquisition to develop before optimizing resource expenditure. On PsyNexus, PRISM achieves an absolute macro-F1 improvement of 5.1% while reducing inference time by 60% relative to full-evidence multimodal supervised fine-tuning. This work provides a methodological foundation for adaptive multimodal intelligence, supporting the development of evidence-seeking agents that reason and make decisions under uncertainty and resource constraints.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Agents, Active Information Acquisition, Multimodal Reasoning, Cost-Aware Reinforcement Learning, Psychiatric Diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_k5KkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PRISM addresses three-way psychiatric diagnosis (MDD, BD, SCZ) as budgeted, partially observed information seeking. An LLM agent with five tools selects clinical sources and EEG analyses; it is trained on roughly 5,000 SFT demonstrations followed by GRPO with a progressive reward moving from grounding to utility to cost. The paper introduces the private PsyNexus cohort (2,051 patients) and reports +5.1 macro-F1 with 60% lower latency versus multimodal SFT on 205 test patients.

Critical strengths

Leakage control is careful and well documented: patient-level partitions, exclusion of test data from retrieval, normalization and hyperparameter selection, and a deployment mode that does not require labels. The ablations are extensive and the scope statement in C.10 is honest about what the results do and do not establish.

Critical weaknesses
  • The headline comparison is not reader-symmetric: the adaptive baselines in Table 2 (LA-CDM 79.0, CDEG 80.5, EDDI 75.0, ACO 76.3) are exactly the SFT-reader column of Fig. S4, so their diagnoses come from a shared SFT reader, while PRISM's 86.3 comes from its own RL policy.
  • Under a common reader, PRISM's acquisition advantage over CDEG is +0.4 or +2.9 rather than +5.8, so the central claim needs to be restated or the comparison re-run; this matters because the paper's selling point is the information-seeking policy, not the reader.
  • The evaluation is thin: no confidence intervals, standard deviations, or significance tests appear; the test set has 205 patients with the BD class at about 14.6% (roughly 30 patients), and macro-F1 is dominated by the minority class, yet no per-class F1 or confusion matrix is reported.
  • The C.13 hyperparameter sweeps are evaluated on the test set.
  • The TDBRAIN probe reports macro-F1 without naming classes, n, or labels, while B.1 itself states TDBRAIN does not provide mutually exclusive patient counts for a three-disorder benchmark, so that result cannot be interpreted.
  • The reported w/o grounding variant (AUBC 0.808) exceeds the full model (0.807), and this is not disclosed in the text.
  • The paper describes no IRB approval, consent procedure, de-identification, or data use agreement for a private psychiatric cohort of 2,051 patients, and states that processed data, including narrative notes, will be made publicly available; given the sensitivity of psychiatric data, this needs resolution before publication.
Questions for authors
  1. Which model produced the baseline diagnoses in Table 2, and what are the numbers under a single common reader for all systems?
  2. Can you report per-class F1 and paired confidence intervals over the 205 test patients?
  3. What are the ethics approval, consent, and de-identification procedures for PsyNexus?
  4. What task, classes, sample size, and labels does the TDBRAIN probe use?
  5. Why is w/o grounding (AUBC 0.808) above the full model (0.807), and what does this imply for the reward design?
4 comments

Thank you for pointing out that the original comparison used different diagnostic readers. We agree that evaluating evidence selection requires holding the reader fixed across policies.

Q1. System-level diagnosis and evidence selection under a common reader

In the submitted Table 2, the SFT reader produced the diagnoses for EDDI, ACO, LA-CDM, and CDEG, whereas PRISM produced its own diagnoses. The original 5.8 percentage-point gain over CDEG (86.3% versus 80.5%) therefore measures an end-to-end system difference. Appendix C.3 (Table 5c) preserves the scores and reader used by each original system. In the revised Table 2b, every policy in a column uses the same frozen diagnostic reader.

Table 2a reports end-to-end diagnosis for the static models, the PESR input controls, and full PRISM. Table 2b reports the six acquisition policies under two shared frozen readers. Within each column, the reader receives only the source evidence selected by each policy. Values are mean macro-F1 (%) over five training seeds on the same 205 test patients.

Acquisition policyShared RL readerShared independent reader
EDDI78.076.5
ACO79.277.8
LA-CDM81.179.8
CDEG82.080.3
PRISM-Base80.078.8
PRISM86.383.2

Against CDEG, the observed acquisition gain was 4.3 percentage points with the shared RL reader, 2.9 with the shared independent reader (paired 95% CI [0.1, 5.9]), and 0.4 with the shared SFT reader (80.9% versus 80.5%, Figure S4). Against LA-CDM, the gains were 5.2 points with the shared RL reader and 3.4 with the independent reader (paired 95% CI [0.2, 6.7]). Sections 4.2 and 4.3 now explain that PRISM-selected evidence yielded higher macro-F1 than CDEG-selected evidence across all three readers, with observed gains ranging from 0.4 to 4.3 points. Table 5a gives paired intervals for the original systems and Table 5b gives them for the shared independent reader. Figure 8b shows both comparisons. The Abstract now reports the independent-reader gain, and the Conclusion gives the range of mean gains across the three shared readers.

Q2. Can you report per-class F1 and paired confidence intervals over the 205 test patients?

The new Figure 8a (Section 4.3, p. 10) shows how PRISM and multimodal SFT perform for each disorder. Appendix C.3 (Table 6) reports end-to-end per-class F1 for all eight methods. For LA-CDM and CDEG, the values are from their original SFT-reader configurations. The results below are five-seed means on the same 205 PsyNexus test patients. SD is the sample standard deviation of macro-F1 across training seeds, in percentage points (pp).

MethodMDD F1 (%)BD F1 (%)SCZ F1 (%)Macro-F1 (%)SD (pp)
EHR-only SFT80.767.283.177.01.1
Multimodal SFT84.374.185.281.21.1
LA-CDM (Bani-Harouni et al., 2026)82.570.384.279.01.3
CDEG (Dai et al., 2026)83.372.785.580.51.2
EHR-only + PESR84.174.385.881.41.5
EHR + EEG metadata + PESR84.575.586.382.11.4
EHR + shuffled EEG + PESR84.374.886.081.71.6
PRISM88.482.088.586.31.1

Relative to multimodal SFT, PRISM gained 4.1 pp in F1 for MDD, 7.9 pp for BD, and 3.3 pp for SCZ. BD showed the largest observed gain and remained the lowest-scoring class for both methods.

The macro-F1 gain over multimodal SFT was 5.1 pp (paired 95% CI [0.9, 9.3]). In the original end-to-end configurations, the gains over LA-CDM and CDEG were 7.3 pp (95% CI [2.6, 12.1]) and 5.8 pp (95% CI [1.4, 10.3]). Q1 above reports the paired intervals for the shared independent reader. For the EHR-only, EEG-metadata, and shuffled-EEG PESR controls, the paired 95% CIs for the macro-F1 gains were [0.8, 9.0], [0.5, 7.9], and [0.7, 8.5] pp, respectively (Table 5). Appendix C.9 describes the 5,000 paired patient-level bootstrap resamples.

Q3. What are the ethics approval, consent, and de-identification procedures for PsyNexus?

For all 2,051 PsyNexus patients, informed consent covered the retrospective use of linked EHR and EEG data for model training and evaluation in this study. Section 3.1 (p. 3) now states the scope of consent. Appendix B.3 (p. 20) documents the ethics review. The parent clinical study received approval from the hospital ethics committee in 2024. The committee reviewed the clinical research protocol and informed consent form (both version 1.1) and approved continuation of the study in 2025 and 2026. We can provide the signed decisions confidentially to the chairs.

The hospital authorized the study team to access PsyNexus data internally under the approved protocol. Hospital policy did not require a separate data use agreement (DUA) for this internal use. We have clarified this basis for access in Appendix B.3 (p. 20). Institutional approval for external sharing of processed patient-level data, including clinical notes, remains pending (Section 3.1, p. 3).

Appendix B.3 (p. 20) now describes how we protected the linked records. We restricted model inputs to predefined fields, used age at EEG acquisition in place of date of birth, and excluded EDF patient-identification fields from the preprocessed EEG signals. We masked names, medical record numbers, and other identifying details in clinical text, checked the masked text with a locally hosted LLM, and manually reviewed a sample. A patient-level assessment considered whether quasi-identifiers and longitudinal patterns could jointly identify an individual. We masked details that posed this risk and retained the temporal relationships needed for clinical reasoning and evidence access.

Q4. What task, classes, sample size, and labels does the TDBRAIN probe use?

For Figure S9, we obtained the class labels from the same TDBRAIN diagnosis field used by XAIguiFormer. We also used the same dataset release and participant split. The EEG-only classification task included 566 participants, of whom 325 had major depressive disorder (MDD), 187 had attention-deficit/hyperactivity disorder (ADHD), and 54 had obsessive-compulsive disorder (OCD). The split comprised 396 training, 85 validation, and 85 test participants. These counts follow Table 5 of XAIguiFormer. The TDBRAIN data descriptor reports EEG sessions by referral indication and availability of a formal diagnosis. van Dijk et al., 2022, Table 1 · Guo et al., ICLR 2025, Table 5

We now report the per-class counts, participant split, and label source in Appendix C.11 (p. 34 and Figure S9 on p. 35). Each probe was fitted on training participants with the EEG encoders frozen. This experiment tests whether explicit EEG measurements complement learned representations under fixed evidence. Appendix C.15 reports the separate end-to-end evaluation of evidence acquisition and diagnosis at a second center.

Q5. Why is w/o grounding (AUBC 0.808) above the full model (0.807), and what does this imply for the reward design?

In Figure S6, removing the grounding reward slightly increases mean AUBC from 0.807 to 0.808. AUBC summarizes diagnostic macro-F1 across acquisition budgets, whereas the grounding reward evaluates whether the acquired evidence supports the report’s claims. Removing grounding reduces citation precision from 96.0% to 86.0% and final macro-F1 from 86.3% to 84.1%. We therefore retain grounding for its contribution to evidence-supported reporting and final diagnostic performance. We have added this interpretation to Section 4.3 (p. 9, lines 475–477) and Appendix C.8 (p. 31, lines 1660–1665), which discusses Figure S6 (p. 32).

Use of the test set in the hyperparameter sensitivity analysis

We selected the stage boundaries and final reward weights on the validation set and fixed them before the test-set sensitivity analysis. Section 4.3 (p. 10, “Hyperparameter sensitivity”) now reports the validation macro-F1, citation precision, and acquisition cost for the selected boundaries. Figures S11 and S12 in Appendix C.13 identify the subsequent test-set sweeps.

To show the validation evidence directly, we reevaluated each configuration's existing checkpoints from five training seeds on the same 205 validation patients. Among nine boundary pairs, the selected pair (0.20, 0.60) attained the highest macro-F1 (87.2%), with 96.4% citation precision and 0.56 normalized cost. Across the reward-weight sweep, increasing the cost coefficient from 0.05 to 0.15 reduced normalized cost from 0.73 to 0.46 and macro-F1 from 87.5% to 86.7%. These results quantify the trade-off between diagnostic performance and acquisition cost around the selected configuration. The additional runs used inference only. Figures S13 and S14 in Appendix C.14 (p. 37) report both validation sweeps.

We hope the revised comparisons make the contribution of evidence selection clearer. We would welcome the opportunity to discuss these results further.

AuthorsOpenReview ↗

Follow-up on our discussion

Dear Reviewer k5Kk,

Thank you for your follow-up comment in the other review thread, where you noted, “I have read the response and have no further questions.” We appreciate your constructive feedback, which helped us clarify the comparisons and strengthen the manuscript.

We hope that the revised manuscript and our clarifications will be helpful as you finalize your assessment. Thank you again for your time and consideration.

Reviewer_DvRSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multimodal psychiatric diagnosis under partial observability and introduces PRISM, an agent framework that jointly selects evidence sources and EEG analysis operations. The authors construct PsyNexus, a cohort of 2,051 patients linking longitudinal electronic health records with EEG recordings. PRISM is initialized through supervised fine-tuning for evidence acquisition and diagnosis, followed by PESR, a progressive reinforcement learning strategy that gradually introduces rewards for evidence grounding and diagnostic utility, along with resource-cost penalties. Experiments show improved diagnostic performance and reduced inference latency compared with full-evidence multimodal supervised fine-tuning. Ablation studies further examine the contributions of evidence selection, representation combinations, and reward scheduling.

Critical strengths
  1. The research problem has practical value. The paper addresses evidence acquisition and analysis selection under resource constraints, allowing the model to use available information to decide which records to examine and which EEG analyses to perform. This provides a meaningful research direction for active multimodal diagnosis.

  2. The experimental design helps distinguish evidence selection from diagnostic capability. The authors evaluate evidence acquired by different strategies using a common frozen diagnostic model. A factorial ablation of source selection and analysis selection further provides direct empirical support for the effectiveness of joint selection.

  3. The progressive reward schedule is supported by a reasonably comprehensive set of comparisons. The paper includes correctness-only rewards, fixed reward weights, and constant weights matched for cumulative coefficient exposure. Evaluations across different optimizers and model backbones further strengthen the empirical evidence for PESR’s effectiveness.

Critical weaknesses
  1. The procedure for establishing reference diagnoses is insufficiently described. Appendix B.3 does not adequately specify the diagnostic criteria used, the reviewers and their qualifications, or the procedures for independent review and disagreement resolution. Differences between diagnoses recorded at the index encounter and the final reference diagnoses are also not reported, making it difficult to determine whether the task primarily involves recovering established diagnoses or predicting diagnoses confirmed later.

  2. The index-encounter selection criteria and information cutoff are not clearly defined. The paper does not specify what makes an encounter eligible or at which stage of the encounter the input cutoff is applied. Consequently, the scope of the EHR and EEG data actually available at the diagnostic decision point is difficult to verify.

  3. The additional benefit of EEG analysis beyond improved EHR retrieval remains unclear. Although the fixed-evidence experiments support the incremental value of EEG-related inputs, the absence of an EHR-only control using the same architecture and PESR training prevents a clear assessment of this additional benefit in the complete system. Moreover, the existing ablations do not adequately disentangle the contribution of EEG signal content from that of recording-quality and availability metadata.

Questions for authors
  1. How were the reference diagnoses established in practice, including the diagnostic criteria, reviewers’ clinical qualifications, and disagreement-resolution procedure? How often did they differ from diagnoses recorded at the index encounter?

  2. What precisely makes an index encounter eligible, and at what stage of that encounter is the information cutoff applied? Which EHR and EEG data are available by that cutoff?

  3. How does PRISM compare with an EHR-only agent using the same architecture and PESR training? Can the contribution of EEG signal content be distinguished from that of recording-quality and availability metadata? Please also clarify whether the EHR-only baseline includes written EEG reports.

4 comments

Thank you for asking us to clarify how the reference diagnoses were established and which records PRISM could access at the index encounter.

Q1. How were the reference diagnoses established in practice, including the diagnostic criteria, reviewers’ clinical qualifications, and disagreement-resolution procedure? How often did they differ from diagnoses recorded at the index encounter?

PsyNexus was supplied to the research team with final diagnoses confirmed by clinical consensus. Three licensed psychiatrists applied ICD-10 criteria and addressed differences in clinical judgment through discussion. The research dataset did not retain each psychiatrist’s initial judgment or a patient-level account of disagreements and their resolution. We therefore cannot reconstruct independent ratings or quantify agreement before consensus. We describe the label source in Section 3.1 and the information available to the research team in Appendix B.3, “Reference diagnoses” (pp. 19–20).

We clarified the eligibility rule in Section 3.1 and give all six screening stages and their counts in Appendix B.3, “Patient eligibility” (p. 19). The EEG dataset annotation and the patient's full available EHR history had to agree on a single target disorder among major depressive disorder (MDD), bipolar disorder (BD), and schizophrenia (SCZ). For example, we excluded a patient with MDD recorded at one encounter and BD at a later encounter, just as we excluded a patient with both diagnoses recorded at the same encounter.

All 205 PsyNexus test patients had a target diagnosis recorded at the index encounter. The target diagnosis in the index record matched the supplied consensus label for 205 of 205 patients (0 mismatches). This count compares the two recorded labels in the selected single-target cohort. The test therefore evaluates whether PRISM can identify an already documented disorder from clinical and EEG evidence after explicit diagnostic labels have been masked. We report the counts in Appendix B.3, “Reference diagnoses” (p. 20) and clarify the scope of the evaluation in Appendix C.10.

Q2. What precisely makes an index encounter eligible, and at what stage of that encounter is the information cutoff applied? Which EHR and EEG data are available by that cutoff?

We evaluated PRISM retrospectively using archived data spanning the full index encounter. For each of the 205 test patients, we selected the earliest encounter meeting the cohort eligibility and EEG linkage criteria. The agent could query eligible EHR records and linked EEG waveforms from that encounter and earlier encounters. We excluded records assigned to subsequent encounters and masked explicit target diagnoses in the evidence and metadata accessible to the agent.

We applied no cutoff within the index encounter. For example, a clinical note documented after EEG acquisition was eligible for retrieval if it belonged to that encounter. This evaluation does not simulate real-time diagnosis at the moment of EEG acquisition or early in a hospital admission. Tool calls reflect the order in which PRISM queried the archive. PsyNexus contained no written EEG reports. The structured EEG examination records used for linkage provided examination dates and encounter identifiers.

We now describe the three matching rules and report how many recordings were linked by each route in Appendix B.3, “EEG-to-EHR encounter linkage” (p. 19). For each recording, we applied the following rules within the same patient.

  1. We matched the waveform’s recording date to the examination date in the same patient’s structured EEG examination records. If all matching records carried the same encounter identifier, we assigned the recording to that encounter.
  2. If there was no same-day examination record, we identified encounters whose date intervals included the EEG recording date. If exactly one interval covered that date, we selected its encounter.
  3. If several intervals covered the date, we selected an encounter only if exactly one of those intervals began on the recording date. We left unresolved recordings unlinked.

In the final cohort, 2,799 EEG recordings were linked to 2,636 distinct encounters. Same-day matches to structured EEG examination records accounted for 2,795 links, and a single covering encounter interval accounted for the other 4. A patient could contribute several encounters, and an encounter could have several recordings. The table below gives the patient-level screening criteria and counts.

Selection stepRetainedCriterion
EEG source deduplication9,284 EEG subject identifiersIdentifiers were deduplicated within the EEG dataset before linkage to the EHR.
Patient linkage3,568 patientsWe required an unambiguous exact match on medical record number.
Demographic verification3,548 patientsWe excluded 20 patients whose recorded gender or date of birth differed across sources.
Encounter linkage3,091 patientsAt least one EEG recording had to link to a specific EHR encounter identifier. This criterion excluded 457 patients.
Target diagnosis agreement2,058 patientsThe sets of target diagnoses in the EEG annotations and the full available EHR history had to be nonempty and identical.
Single target diagnosis2,051 patientsThe EEG cohort label had to match the sole target diagnosis in the full available EHR history. Of the 2,058 patients with matching label sets, seven had multiple target diagnoses and were excluded at this step.

The final cohort comprised 947 patients with MDD, 299 with BD, and 805 with SCZ (Section 3.1). We clarified the encounter boundary in Section 3.1, “Diagnostic episodes,” and Appendix B.3, “Reference diagnoses” (p. 20). The paragraph “Scope of the evaluation” in Appendix C.10 (p. 33) explains what can be inferred from this retrospective design.

Q3. Contribution of EEG signal content beyond EHR and recording metadata

The EHR-only + PESR control in Table 2a used the same Qwen3.5-9B backbone and policy architecture as full PRISM. Its evidence access was restricted to clinical records. Full PRISM could additionally analyze EEG waveforms and obtain measurements and representations derived from them. PsyNexus contained no written EEG reports. The structured EEG examination records supplied dates and encounter identifiers for linkage (Appendix B.3). We clarified the inputs in Section 4.2 and Appendix C.3 (p. 28, “EEG input controls”).

We independently trained the EHR-only, EEG-metadata, and shuffled-EEG controls on the same patients with the same PESR rewards and acquisition budget as full PRISM. On the 205 PsyNexus test patients, macro-F1 was 81.4% for EHR-only PESR and 86.3% for full PRISM. The metadata control retained recording metadata and excluded signal content. The shuffled control preserved the distribution of EEG signals while disrupting patient correspondence. The table reports paired macro-F1 gains on PsyNexus and at the second center, where full PRISM achieved 79.0%.

ComparatorPRISM gain on PsyNexus, pp [95% CI]PRISM gain at the second center, pp [95% CI]
EHR-only + PESR+4.9 [0.8, 9.0]+4.4 [0.5, 8.6]
EHR + EEG metadata + PESR+4.2 [0.5, 7.9]+3.7 [0.2, 7.5]
EHR + shuffled EEG + PESR+4.6 [0.7, 8.5]+4.2 [0.4, 8.2]

The paired macro-F1 differences support an incremental contribution from patient-aligned EEG beyond recording metadata. The same pattern appeared in the preliminary zero-shot evaluation of 240 patients at the second center. Table 2a reports the PsyNexus end-to-end scores, and Section 4.2 (“Incremental value of EEG”) gives the three paired intervals. Appendix C.3 (Table 5a, p. 27) reports the full differences, and Appendix C.15 (Table 7a, p. 38) reports the external comparisons.

Ethics review. Research approval and proposed data sharing

For all 2,051 PsyNexus patients, informed consent covered the retrospective use of linked EHR and EEG data for model training and evaluation in this study. We now describe consent in Section 3.1 (p. 3) and the ethics review in Appendix B.3 (p. 20). The hospital ethics committee approved the parent clinical study in 2024 after reviewing the protocol and informed consent form. It subsequently approved continuation in 2025 and 2026. We can provide the signed decisions confidentially to the chairs. We have added our de-identification procedures to Appendix B.3 (p. 20), “De-identification and re-identification risk.” We masked identifying details in clinical narratives, checked the masked text with a locally hosted LLM, and manually reviewed a sample of records. We assessed whether combinations of quasi-identifiers and longitudinal patterns could identify a patient across linked EHR and EEG records, and masked overly specific details while preserving the timing relationships needed to interpret the evidence. We have revised Section 3.1 (p. 3) to state that external sharing of processed patient-level PsyNexus data for non-commercial academic research awaits institutional approval.

We would be happy to discuss any remaining questions about the cohort or the retrospective evaluation.

AuthorsOpenReview ↗

Follow-up on our revisions

Dear Reviewer DvRS,

Thank you again for your helpful feedback. Our revised manuscript clarifies the reference diagnoses, retrospective evaluation setting, and data governance (Appendix B.3), and includes additional EEG controls (Table 2a and Appendix C.3).

We hope these revisions clarify the evaluation setting and the contribution of EEG, and would welcome any further questions or suggestions.

Reviewer_3DsQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PRISM studies psychiatric diagnosis through sequential acquisition of electronic health records (EHR) and electroencephalography (EEG) evidence. The policy selects both evidence sources and analysis operations, using staged reinforcement learning rewards for diagnosis, grounding, information gain, and cost. Experiments use PsyNexus, a cohort of 2,051 patients, and report improved diagnostic performance and acquisition efficiency over static and adaptive baselines.

Critical strengths
  • Jointly selecting evidence sources and EEG analysis operations gives the acquisition problem a concrete, clinically relevant structure. The independent-reader evaluation helps separate acquisition quality from the final diagnostic model.
  • The controls are more extensive than a simple comparison against static fusion. Matched training exposure, reward schedules, and acquisition budgets address several plausible explanations for the reported gains.
Critical weaknesses
  • The incremental value of EEG under the final training procedure remains unclear. The EHR-only comparator in Table 2 uses supervised fine-tuning, whereas PRISM receives the full reinforcement learning procedure. The EEG feature and acquisition ablations are useful, but do not replace an EHR-only policy trained with the same rewards and budget. This matters when attributing the improvement to multimodal reasoning.
  • The main performance claims rely on five-seed means evaluated on the same 205 test patients, without uncertainty estimates for the differences. Repeated training does not increase the number of independent test patients. The external EEG probes assess representations rather than the complete diagnostic agent. The magnitude and portability of the end-to-end gain therefore remain insufficiently established.
Questions for authors
  • How does an EHR-only policy perform when trained with the same reinforcement learning procedure, data exposure, and acquisition budget as PRISM?
  • Can you report paired patient-bootstrap confidence intervals for the main performance differences, together with variation across the five training seeds?
  • Which parts of the complete acquisition and diagnosis pipeline have been validated outside PsyNexus, beyond the external EEG representation probes?
  • How many clinicians adjudicated the diagnostic labels, how were disagreements resolved, and what checks verified that masked diagnostic information was absent from accessible records?
7 comments

Thank you for your careful review. Your questions prompted us to examine how much patient-matched EEG contributes and whether the diagnostic gain extends to a second center.

Q1. How much does EEG contribute under the same reinforcement learning procedure?

We trained three input controls on the same patients with the same PESR rewards and budget as PRISM. On the same 205 test patients, full PRISM achieved 86.3% macro-F1, exceeding EHR-only PESR (81.4%) by 4.9 percentage points. EHR with EEG metadata yielded 82.1%, and EHR with EEG shuffled across patients yielded 81.7%. These controls support a contribution from the patient's own EEG beyond clinical records and recording metadata. Table 2a reports end-to-end performance for all three input controls. Section 4.2 (“Incremental value of EEG”) gives the corresponding paired patient-bootstrap intervals.

Q2. How uncertain are the performance differences across patients and across training seeds?

On the same 205 test patients, PRISM achieved 86.3% macro-F1 with a sample SD of 1.1 percentage points (pp) across five training seeds. Its gain over multimodal SFT was 5.1 pp (paired 95% CI [0.9, 9.3]). Section 4.2 (p. 7) now reports this interval alongside the five-seed SD of both PRISM and multimodal SFT. Appendix C.3 (Tables 5 and 6) gives the six end-to-end comparisons below. For LA-CDM and CDEG, the SFT reader classified the selected evidence, while PRISM produced its own report (original configurations, Table 5c).

ComparatorMean macro-F1 (%)Seed SD (pp)PRISM gain (pp)Paired 95% CI (pp)
Multimodal SFT81.21.1+5.1[0.9, 9.3]
EHR-only + PESR81.41.5+4.9[0.8, 9.0]
EHR + EEG metadata + PESR82.11.4+4.2[0.5, 7.9]
EHR + shuffled EEG + PESR81.71.6+4.6[0.7, 8.5]
LA-CDM79.01.3+7.3[2.6, 12.1]
CDEG80.51.2+5.8[1.4, 10.3]

We also compared the evidence selected by each policy under one frozen independent reader. The reader achieved 83.2% macro-F1 with PRISM-selected evidence, 79.8% with LA-CDM, and 80.3% with CDEG. The paired gains were 3.4 pp [0.2, 6.7] over LA-CDM and 2.9 pp [0.1, 5.9] over CDEG (95% CIs, Table 5b). Both intervals favor PRISM-selected evidence while allowing for a small gain. Table 2b reports macro-F1 under the shared RL and independent readers. Figure 8b (Section 4.3) displays the independent-reader intervals alongside the intervals for the original end-to-end comparisons.

For each of 5,000 bootstrap resamples, we drew 205 patients with replacement, used the same patient indices for both methods and all seeds, and averaged the five seed-specific macro-F1 differences (Appendix C.9). The SDs in Table 6 quantify training-seed variability on the full test cohort.

Q3. Which stages of acquisition and diagnosis have been evaluated outside PsyNexus?

We conducted a zero-shot cross-center evaluation of the complete PRISM pipeline on 240 patients with paired EHR and EEG at a second center (98 SCZ, 88 MDD, 54 BD). PRISM selected clinical records and EEG analyses and issued a final diagnosis. The models, decision thresholds, and hyperparameters were fixed on PsyNexus before external evaluation.

End-to-end diagnosis. PRISM achieved 79.0% macro-F1 across all 240 patients, compared with 73.4% for full-evidence multimodal SFT. The gain was 5.6 percentage points (paired patient-bootstrap 95% CI [1.1, 10.0], Table 7a). The corresponding gain on PsyNexus was 5.1 points (Table 2a).

Evidence acquisition. A shared frozen independent reader achieved 2.5 percentage points higher macro-F1 with PRISM-selected evidence than with CDEG-selected evidence (76.6% versus 74.1%). The same reader achieved 70.8% with fixed-order acquisition (Table 7b).

Appendix C.15 (p. 38) reports these preliminary results from one external center. Future work will assess how the diagnostic and evidence-selection gains vary across centers.

Q4. Who adjudicated the reference diagnoses, and how was target-label masking verified?

The PsyNexus labels we received had already been confirmed by clinical consensus. Three licensed psychiatrists applied ICD-10 criteria and discussed differing opinions to establish the final diagnosis. The research dataset contained no patient-level record of initial disagreements or how they were resolved, so we cannot quantify disagreement before consensus. We clarify the label source in Section 3.1 and Appendix B.3, “Reference diagnoses.”

We screened structured diagnosis fields and clinical narratives for explicit diagnostic terms, their synonyms, and corresponding ICD codes. We manually reviewed potentially revealing content and inspected the records and source metadata returned by the agent's tools for explicit target diagnoses. We found no explicit disclosure of the target diagnosis in the accessible evidence we examined. Clinical observations, including symptoms, treatment histories, and assessment findings, were retained when they did not explicitly disclose the diagnosis. Appendix B.3 (p. 20, “Target-label masking audit”) describes these checks.

Ethics review. Institutional approval and data governance

The approved protocol covered the retrospective linkage of each patient’s EHR and EEG data to train and evaluate diagnostic models. The hospital ethics committee approved the parent clinical study in May 2024 after reviewing the protocol and informed consent form (both version 1.1). The committee approved continuation in 2025 and 2026. Informed consent covered this use for all 2,051 PsyNexus patients. We now specify the scope of the approved protocol in Appendix B.3 (p. 20) and the scope of consent in Section 3.1 (p. 3) and Appendix B.3. We can provide the signed decisions and approval identifiers confidentially to the chairs.

We have added the de-identification procedure and the patient-level re-identification risk assessment to Appendix B.3 (p. 20), “De-identification and re-identification risk.” After linking EHR and EEG records locally, we used stable study identifiers and restricted access to the index linking them to the original identifiers. We masked names, medical record numbers, and other identifying details in clinical text. A locally hosted LLM checked the masked text for identifying information, and we manually reviewed a sample of records.

We also examined whether quasi-identifiers combined with longitudinal patterns across a patient's linked records could identify that patient. We masked identifying or overly specific details while preserving the temporal relationships needed to interpret clinical events and distinguish earlier from later encounters. We have revised Section 3.1 (p. 3) to clarify that institutional approval for external sharing of processed patient-level PsyNexus data is pending. Any future release would be limited to the fields and access conditions authorized by the institution.

We would welcome your thoughts on what the second-center results establish and where further validation would be most informative.

Thanks for the clarifications on the common-reader comparison, the EEG contribution, and the ethics documentation. I have read the response and have no further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.