- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
PRISM addresses three-way psychiatric diagnosis (MDD, BD, SCZ) as budgeted, partially observed information seeking. An LLM agent with five tools selects clinical sources and EEG analyses; it is trained on roughly 5,000 SFT demonstrations followed by GRPO with a progressive reward moving from grounding to utility to cost. The paper introduces the private PsyNexus cohort (2,051 patients) and reports +5.1 macro-F1 with 60% lower latency versus multimodal SFT on 205 test patients.
Critical strengths
Leakage control is careful and well documented: patient-level partitions, exclusion of test data from retrieval, normalization and hyperparameter selection, and a deployment mode that does not require labels. The ablations are extensive and the scope statement in C.10 is honest about what the results do and do not establish.
Critical weaknesses
- The headline comparison is not reader-symmetric: the adaptive baselines in Table 2 (LA-CDM 79.0, CDEG 80.5, EDDI 75.0, ACO 76.3) are exactly the SFT-reader column of Fig. S4, so their diagnoses come from a shared SFT reader, while PRISM's 86.3 comes from its own RL policy.
- Under a common reader, PRISM's acquisition advantage over CDEG is +0.4 or +2.9 rather than +5.8, so the central claim needs to be restated or the comparison re-run; this matters because the paper's selling point is the information-seeking policy, not the reader.
- The evaluation is thin: no confidence intervals, standard deviations, or significance tests appear; the test set has 205 patients with the BD class at about 14.6% (roughly 30 patients), and macro-F1 is dominated by the minority class, yet no per-class F1 or confusion matrix is reported.
- The C.13 hyperparameter sweeps are evaluated on the test set.
- The TDBRAIN probe reports macro-F1 without naming classes, n, or labels, while B.1 itself states TDBRAIN does not provide mutually exclusive patient counts for a three-disorder benchmark, so that result cannot be interpreted.
- The reported w/o grounding variant (AUBC 0.808) exceeds the full model (0.807), and this is not disclosed in the text.
- The paper describes no IRB approval, consent procedure, de-identification, or data use agreement for a private psychiatric cohort of 2,051 patients, and states that processed data, including narrative notes, will be made publicly available; given the sensitivity of psychiatric data, this needs resolution before publication.
Questions for authors
- Which model produced the baseline diagnoses in Table 2, and what are the numbers under a single common reader for all systems?
- Can you report per-class F1 and paired confidence intervals over the 205 test patients?
- What are the ethics approval, consent, and de-identification procedures for PsyNexus?
- What task, classes, sample size, and labels does the TDBRAIN probe use?
- Why is w/o grounding (AUBC 0.808) above the full model (0.807), and what does this imply for the reward design?
4 comments
Q3. What are the ethics approval, consent, and de-identification procedures for PsyNexus?
For all 2,051 PsyNexus patients, informed consent covered the retrospective use of linked EHR and EEG data for model training and evaluation in this study. Section 3.1 (p. 3) now states the scope of consent. Appendix B.3 (p. 20) documents the ethics review. The parent clinical study received approval from the hospital ethics committee in 2024. The committee reviewed the clinical research protocol and informed consent form (both version 1.1) and approved continuation of the study in 2025 and 2026. We can provide the signed decisions confidentially to the chairs.
The hospital authorized the study team to access PsyNexus data internally under the approved protocol. Hospital policy did not require a separate data use agreement (DUA) for this internal use. We have clarified this basis for access in Appendix B.3 (p. 20). Institutional approval for external sharing of processed patient-level data, including clinical notes, remains pending (Section 3.1, p. 3).
Appendix B.3 (p. 20) now describes how we protected the linked records. We restricted model inputs to predefined fields, used age at EEG acquisition in place of date of birth, and excluded EDF patient-identification fields from the preprocessed EEG signals. We masked names, medical record numbers, and other identifying details in clinical text, checked the masked text with a locally hosted LLM, and manually reviewed a sample. A patient-level assessment considered whether quasi-identifiers and longitudinal patterns could jointly identify an individual. We masked details that posed this risk and retained the temporal relationships needed for clinical reasoning and evidence access.
Q4. What task, classes, sample size, and labels does the TDBRAIN probe use?
For Figure S9, we obtained the class labels from the same TDBRAIN diagnosis field used by XAIguiFormer. We also used the same dataset release and participant split. The EEG-only classification task included 566 participants, of whom 325 had major depressive disorder (MDD), 187 had attention-deficit/hyperactivity disorder (ADHD), and 54 had obsessive-compulsive disorder (OCD). The split comprised 396 training, 85 validation, and 85 test participants. These counts follow Table 5 of XAIguiFormer. The TDBRAIN data descriptor reports EEG sessions by referral indication and availability of a formal diagnosis. van Dijk et al., 2022, Table 1 · Guo et al., ICLR 2025, Table 5
We now report the per-class counts, participant split, and label source in Appendix C.11 (p. 34 and Figure S9 on p. 35). Each probe was fitted on training participants with the EEG encoders frozen. This experiment tests whether explicit EEG measurements complement learned representations under fixed evidence. Appendix C.15 reports the separate end-to-end evaluation of evidence acquisition and diagnosis at a second center.
Q5. Why is w/o grounding (AUBC 0.808) above the full model (0.807), and what does this imply for the reward design?
In Figure S6, removing the grounding reward slightly increases mean AUBC from 0.807 to 0.808. AUBC summarizes diagnostic macro-F1 across acquisition budgets, whereas the grounding reward evaluates whether the acquired evidence supports the report’s claims. Removing grounding reduces citation precision from 96.0% to 86.0% and final macro-F1 from 86.3% to 84.1%. We therefore retain grounding for its contribution to evidence-supported reporting and final diagnostic performance. We have added this interpretation to Section 4.3 (p. 9, lines 475–477) and Appendix C.8 (p. 31, lines 1660–1665), which discusses Figure S6 (p. 32).
Use of the test set in the hyperparameter sensitivity analysis
We selected the stage boundaries and final reward weights on the validation set and fixed them before the test-set sensitivity analysis. Section 4.3 (p. 10, “Hyperparameter sensitivity”) now reports the validation macro-F1, citation precision, and acquisition cost for the selected boundaries. Figures S11 and S12 in Appendix C.13 identify the subsequent test-set sweeps.
To show the validation evidence directly, we reevaluated each configuration's existing checkpoints from five training seeds on the same 205 validation patients. Among nine boundary pairs, the selected pair (0.20, 0.60) attained the highest macro-F1 (87.2%), with 96.4% citation precision and 0.56 normalized cost. Across the reward-weight sweep, increasing the cost coefficient from 0.05 to 0.15 reduced normalized cost from 0.73 to 0.46 and macro-F1 from 87.5% to 86.7%. These results quantify the trade-off between diagnostic performance and acquisition cost around the selected configuration. The additional runs used inference only. Figures S13 and S14 in Appendix C.14 (p. 37) report both validation sweeps.
We hope the revised comparisons make the contribution of evidence selection clearer. We would welcome the opportunity to discuss these results further.
Follow-up on our discussion
Dear Reviewer k5Kk,
Thank you for your follow-up comment in the other review thread, where you noted, “I have read the response and have no further questions.” We appreciate your constructive feedback, which helped us clarify the comparisons and strengthen the manuscript.
We hope that the revised manuscript and our clarifications will be helpful as you finalize your assessment. Thank you again for your time and consideration.
Thank you for pointing out that the original comparison used different diagnostic readers. We agree that evaluating evidence selection requires holding the reader fixed across policies.
Q1. System-level diagnosis and evidence selection under a common reader
In the submitted Table 2, the SFT reader produced the diagnoses for EDDI, ACO, LA-CDM, and CDEG, whereas PRISM produced its own diagnoses. The original 5.8 percentage-point gain over CDEG (86.3% versus 80.5%) therefore measures an end-to-end system difference. Appendix C.3 (Table 5c) preserves the scores and reader used by each original system. In the revised Table 2b, every policy in a column uses the same frozen diagnostic reader.
Table 2a reports end-to-end diagnosis for the static models, the PESR input controls, and full PRISM. Table 2b reports the six acquisition policies under two shared frozen readers. Within each column, the reader receives only the source evidence selected by each policy. Values are mean macro-F1 (%) over five training seeds on the same 205 test patients.
Against CDEG, the observed acquisition gain was 4.3 percentage points with the shared RL reader, 2.9 with the shared independent reader (paired 95% CI [0.1, 5.9]), and 0.4 with the shared SFT reader (80.9% versus 80.5%, Figure S4). Against LA-CDM, the gains were 5.2 points with the shared RL reader and 3.4 with the independent reader (paired 95% CI [0.2, 6.7]). Sections 4.2 and 4.3 now explain that PRISM-selected evidence yielded higher macro-F1 than CDEG-selected evidence across all three readers, with observed gains ranging from 0.4 to 4.3 points. Table 5a gives paired intervals for the original systems and Table 5b gives them for the shared independent reader. Figure 8b shows both comparisons. The Abstract now reports the independent-reader gain, and the Conclusion gives the range of mean gains across the three shared readers.
Q2. Can you report per-class F1 and paired confidence intervals over the 205 test patients?
The new Figure 8a (Section 4.3, p. 10) shows how PRISM and multimodal SFT perform for each disorder. Appendix C.3 (Table 6) reports end-to-end per-class F1 for all eight methods. For LA-CDM and CDEG, the values are from their original SFT-reader configurations. The results below are five-seed means on the same 205 PsyNexus test patients. SD is the sample standard deviation of macro-F1 across training seeds, in percentage points (pp).
Relative to multimodal SFT, PRISM gained 4.1 pp in F1 for MDD, 7.9 pp for BD, and 3.3 pp for SCZ. BD showed the largest observed gain and remained the lowest-scoring class for both methods.
The macro-F1 gain over multimodal SFT was 5.1 pp (paired 95% CI [0.9, 9.3]). In the original end-to-end configurations, the gains over LA-CDM and CDEG were 7.3 pp (95% CI [2.6, 12.1]) and 5.8 pp (95% CI [1.4, 10.3]). Q1 above reports the paired intervals for the shared independent reader. For the EHR-only, EEG-metadata, and shuffled-EEG PESR controls, the paired 95% CIs for the macro-F1 gains were [0.8, 9.0], [0.5, 7.9], and [0.7, 8.5] pp, respectively (Table 5). Appendix C.9 describes the 5,000 paired patient-level bootstrap resamples.