- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper has two main contributions: 1) It introduces EpiKG, an epilepsy knowledge graph built from seven ontology sources and 48,166 PubMed papers; 2) It also introduces the EpiBench benchmark, which includes five tasks (T1 clinical QA, T2 clinical report generation from EEG and patient information, T3 biomarker-driven precision medicine, T4 treatment recommendation and T5 deep research planning). The Graph-RAG retriever, which leverages the EpiKG, is compared with several baselines and shows improved performance, as compared to the baselines, on most tasks.
Critical strengths
The knowledge graph constructed addresses a critical gap in the epilepsy literature: existing epilepsy resources (EpSO, Epilepsy Ontology) are annotation vocabularies and do not support chains such as gene → mechanism → drug → contraindication.
Schema grounded in authoritative sources (ILAE, OMIM, HPO, ChEBI, AES, CPIC).
Broad benchmark for extrinsic evaluation: QA, generation, precision medicine, treatment, and research planning in one package, including a workflow-realistic EEG-to-impression task on real neurologist notes.
Hyperparameters seem to be "selected on T1 MCQ validation performance," but Table 7 lists every split as test-only.
Critical weaknesses
The most significant limitation is the lack of validation of the EpiKG quality. 70% of triplets are LLM-extracted, yet there is no precision estimate, no sampled audit, no inter-annotator agreement for the expert review of flagged candidates.
T3 gold comes from CPIC/ILAE; the KG is seeded from some of those guidelines. Consequently, the Graph-RAG gains may measure answer lookup/graph adjacency rather than reasoning. There is no retrieval-only (no-LLM) baseline, and no per-category T3 breakdown. Those would be useful to show.
Baselines such as MedRAG/DR.KNOWS/AMG-RAG are under-reported and their corresponding backbone LLM, task, and metric are unspecified.
Questions for authors
What is the measured precision of LLM-extracted triplets on a random audited sample? How many candidates reached expert review and what fraction survived?
On which split were the Fig. 3 hyperparameters chosen?