- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose MedRDSyn a knowledge-graph–guided framework for addressing the scarcity of real clinical records for rare diseases. The framework constructs a rare disease knowledge graph by integrating structured biomedical databases and unstructured text. It employs multiple-source triple generation, hierarchical multi-source knowledge fusion framework which includes entity alignment, ontology-based validation, and LLM-assisted relation weighting to improve the reliability of extracted knowledge. MedRDSyn then extracts relevant subgraphs to guide the generation of synthetic rare disease clinical cases. The resulting synthetic dataset, MedRDSyn-14k, is used for SFT of LLMs. The authors evaluate the approach on medical QA benchmarks.
Critical strengths
- The authors introduce a framework for constructing a rare disease-oriented KG containing 48,379 entities and 654,814 triples.
- The authors also introduce MedRDSyn-14k, a dataset containing 14,035 synthetic clinical samples that can be used to fine-tune medical LLMs, particularly for rare diseases where real clinical data is scarce. Unlike approaches that mainly rely on disease–phenotype associations, MedRDSyn incorporates pathogenic chains to guide the generation of more biologically plausible clinical records.
Critical weaknesses
-
A major concern is the substantial methodological and engineering complexity introduced by MedRDSyn relative to the reported performance gains. The framework requires multiple processing stages, including entity recognition, relation extraction, entity alignment, knowledge fusion, LLM-based relation weighting, PageRank computation, subgraph extraction and synthetic data generation etc… Several of these stages rely on additional LLM calls, potentially introducing considerable computational overhead. However, the reported performance gains appear relatively modest, with average improvements ranging from approximately +0.48 to +0.84 percentage points. Given the complexity of the proposed framework, the paper would benefit from a more thorough discussion of whether the performance improvements justify the additional computational and engineering costs.
-
Some individual improvements are particularly small, such as Qwen3-0.6B on MedQA with +0.12 or Qwen3-8B on PubMedQA with +0.20 points. Without statistical significance testing, it is difficult to determine whether these improvements are robust or fall within normal experimental variability.
-
The paper does not provide a sufficiently detailed analysis of the computational resources required to construct and use MedRDSyn. How much additional computational cost is introduced by: 1) Knowledge graph construction (+LLM usage) 2) Subgraph extraction and synthetic clinical case generation and 3) Additional supervised fine-tuning. A comparison of GPU-hours, inference costs, API usage, and total execution time would help assess the practical efficiency and scalability of the framework.
-
The authors state: "By constructing a rare disease-oriented KG and retrieving pathogenic-chain subgraphs, MedRDSyn explicitly supplements long-tail disease supervision signals and therefore continues to provide gains for larger models." However, the presented results do not directly demonstrate that the improvements are specifically attributable to better performance on rare or long-tail diseases. Indeed, the evaluation primarily only reports aggregate performance on general medical QA datasets. A more convincing evaluation would stratify benchmark questions according to disease prevalence or frequency in the training data, distinguishing between common and rare diseases. The authors could then investigate whether MedRDSyn provides significantly larger improvements for rare disease-related questions. Such an analysis would directly support the central motivation of the paper and demonstrate whether the proposed framework is particularly beneficial for long-tail medical knowledge.
-
The proposed knowledge graph is constructed through a largely automated pipeline. Although the authors introduce mechanisms intended to improve the reliability of extracted relationships, these mechanisms do not guarantee factual correctness. The paper would benefit from a quantitative human evaluation of a representative sample of graph triples. This is particularly important because errors in the KG may propagate into synthetic clinical records.
Questions for authors
-
In the related work section, the authors state: "Early KGs relied on expert construction (Smith et al., 2007), while recent automated pipelines construct disease-specific or rare disease KGs from structured resources and task-specific evidence graphs (Karki et al., 2025; Zhu et al., 2020; Qin et al., 2025). Nevertheless, existing automated rare disease KGs still face challenges such as data sparsity, relation noise, and constraint-based validation (Hogan et al., 2021)." Could the authors clarify how MedRDSyn specifically addresses the “relation noise and constraint-based validation” limitations?
-
How were the expert-validated clinical templates constructed? Could the authors clarify: How many experts participated, and what were their clinical qualifications? How were the experts recruited?
-
Could the authors justify their decision to use GPT-5.2 rather than specialized biomedical language models, such as BioBERT or PubMedBERT, for entity recognition and relation extraction? Previous studies, including “Benchmarking Large Language Models for Biomedical Natural Language Processing Applications and Recommendations” and “Exploring Transformer Models: Fine-Tuning vs. Inference on Relation Extraction from Biomedical Texts” report that specialized biomedical models can outperform general-purpose LLMs for biomedical relation extraction tasks.
-
Since the HMKF framework relies heavily on automated entity recognition, relation extraction, and LLM-based validation, errors may remain in the final graph. For example, some relations can be wrong either because entities are wrong or because the relation is false. Did the authors conduct a human evaluation of a representative sample of graph entities, relations, or pathogenic chains?
-
The authors primarily evaluate MedRDSyn through downstream medical question-answering benchmarks. Given that MedRDSyn is designed to address rare disease data scarcity, other downstream applications than QA could provide more direct evidence of its effectiveness. (e.g. Disease classification from clinical notes). Why is the downstream evaluation limited to medical Q/A?
5 comments
- Could the authors justify their decision to use GPT-5.2 rather than specialized biomedical language models, such as BioBERT or PubMedBERT, for entity recognition and relation extraction? Previous studies, including “Benchmarking Large Language Models for Biomedical Natural Language Processing Applications and Recommendations” and “Exploring Transformer Models: Fine-Tuning vs. Inference on Relation Extraction from Biomedical Texts” report that specialized biomedical models can outperform general-purpose LLMs for biomedical relation extraction tasks.
GPT-5.2 performs ontology-guided structured knowledge extraction to support rare disease clinical record synthesis, with entity recognition as one component. We first design the schema around the information needed for record generation. It covers patient information, symptoms and signs, diagnostic examinations, and therapeutic interventions in clinical records, while additionally incorporating genes, variants, proteins, phenotypes, and their connecting relations to organize rare disease pathogenic mechanisms. This yields unified definitions for 21 entity types and 28 relation types. GPT-5.2 extracts knowledge from unstructured literature under this schema. The model must interpret the literature according to the entity and relation definitions, assign mentions to the appropriate types, establish relations between entities, record relation attributes such as diagnostic findings, pathogenic mechanisms, and drug interaction endpoints, and retain the corresponding evidence from the source text. Together, these fields specify the medical meaning of a relation.
This task requires a variable number of entities, relations, and associated fields to be generated under unified ontology constraints, making it suitable for a generative language model with instruction-following capabilities. A decoder model can organize these interconnected fields into a unified structured output, retaining relations alongside their specific medical descriptions and evidence within the same extraction process. By comparison, encoder models such as BioBERT and PubMedBERT typically perform extraction through entity tagging or relation classification heads. Adapting them to our full output requirements would additionally require modules for attribute and evidence extraction.
These requirements make context preservation an important consideration in model selection. Relevant medical descriptions may span adjacent sentences or paragraphs, requiring context to resolve entity references, relations, and their applicable conditions. Both BioBERT [1] and PubMedBERT [2] are based on the BERT-BASE architecture. Their standard models use learned absolute positional embeddings and support input sequences of up to 512 tokens without positional encoding extension. GPT-5.2 provides a longer context window, allowing more room for ontology definitions, more complete literature passages, and structured output. Extraction results subsequently undergo entity normalization, relation validation, and multi-source evidence fusion. The retained relation attributes help distinguish and aggregate relations, while the source text evidence is provided with retrieved subgraphs to support subsequent clinical record and QA+CoT generation. The goal of this stage is therefore to provide structured knowledge with specific medical meaning and traceable literature evidence for record synthesis. As a general-purpose decoder model, GPT-5.2 is better suited to this task format.
[1] Lee J, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020;36(4):1234–1240. doi:10.1093/bioinformatics/btz682.
[2] Gu Y, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare. 2022;3(1):Article 2. doi:10.1145/3458754.
- Since the HMKF framework relies heavily on automated entity recognition, relation extraction, and LLM-based validation, errors may remain in the final graph. For example, some relations can be wrong either because entities are wrong or because the relation is false. Did the authors conduct a human evaluation of a representative sample of graph entities, relations, or pathogenic chains?
In addition to automated knowledge graph validation, we apply quality control to clinical records generated from combined graph knowledge before training. Rule-based filtering, overall quality assessment, and manual spot-checks are used to control the risk of potential errors entering the subsequent training data.
Rule-based filtering removes failed generations, empty texts, erroneous outputs, and records with abnormal lengths. Following Appendix C.3, GPT-OSS performs an overall quality assessment, focusing on content coherence, the clinical plausibility of the patient scenario, clearly unsafe or implausible descriptions, and the completeness of key clinical content. This step assesses the overall usability of graph knowledge once it has been combined into a specific patient scenario. Even when individual relations pass earlier checks, their integration into the medical history, examinations, diagnosis, and treatment course still requires assessment at the record level.
After automated filtering, we further review retained records through manual spot-checks. Appendix C.3 defines a score of 4 as good overall quality with minor issues, and a score of 5 as coherent content, a clinically plausible scenario, and complete key content. To allocate more expert review effort to samples that may still contain minor issues, we set sampling weights according to the automated scores: each record scored 4 receives three times the sampling weight of a record scored 5. This allows us to check for problems that automated assessment may have missed even in highly rated records.
Manual spot-checks use medical content validity as a single binary criterion, distinguishing pass from fail: 1 (pass) indicates medically accurate content with no obvious errors or severe medical implausibilities; 0 (fail) indicates clear medical errors or seriously implausible medical statements. The assessment focuses on whether the medical content meets basic validity requirements.
In this process, automated scoring filters out records with clear problems and prioritizes records for subsequent manual spot-checks. Manual spot-checks complement automated assessment by examining whether the retained records still contain obvious medical errors or seriously implausible content.
- The authors primarily evaluate MedRDSyn through downstream medical question-answering benchmarks. Given that MedRDSyn is designed to address rare disease data scarcity, other downstream applications than QA could provide more direct evidence of its effectiveness. (e.g. Disease classification from clinical notes). Why is the downstream evaluation limited to medical Q/A?
We include MedQA, PubMedQA, and ReDisQA to assess general medical knowledge and rare disease knowledge after training on the synthetic data. MedQA assesses clinical medical knowledge and its application to problem scenarios; PubMedQA assesses comprehension and judgment of evidence from biomedical literature; and ReDisQA assesses rare disease knowledge and reasoning. We also use RareBench to evaluate performance on disease prediction [3].
RareBench Task 4 evaluates phenotype-driven differential diagnosis of rare diseases. Given a patient's phenotypes or symptoms, the model generates and ranks the most likely candidate diseases. It must connect multiple patient findings with disease characteristics, distinguish between diseases that may have similar phenotypes, and assign priorities to the candidate diagnoses. The R@1 metric we report requires the model to identify the correct disease and rank it as the most likely diagnosis, reflecting both disease identification and the ability to rank the correct diagnosis first in differential diagnosis. RareBench thus evaluates prediction from patient clinical features to a disease diagnosis through open-ended generation, allowing us to examine the effect of synthetic training data on rare disease identification and diagnostic ability.
[3] Chen X, et al. RareBench: Can LLMs Serve as Rare Diseases Specialists? In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024:4850–4861. doi:10.1145/3637528.3671576.
We thank the reviewer for the careful review. Below, we explain the design of knowledge validation and clinical record quality control, the construction of clinical templates, the choice of extraction model, and the evaluation settings for rare disease diagnosis.
Our improvements address how knowledge from multiple sources is validated and used for clinical record synthesis. In an automatically constructed graph, a relation may satisfy type constraints while still lacking sufficient medical support. If such relations are treated equally during generation, noise retained during graph construction may propagate into clinical records and reasoning explanations.
We therefore apply quality control across knowledge fusion and generation. The schema, defined around clinical record requirements, enables relations from different sources to be checked against unified entity roles and connection rules. We then further assess the medical semantic reliability of relations that pass fusion and constraint checks. For each triple, the system retains the head and tail entities and constructs a positive statement from the original relation and a negative statement from its negation. GPT-OSS assesses the correctness of each statement and assigns a confidence score. By combining support for the positive statement with rejection of the negative statement, the system obtains a semantic consistency score for the relation, adding a reliability assessment beyond type constraints. This score is then combined with the source confidence obtained during fusion to produce the final relation weight. The weight contributes to node importance estimation and candidate relation ranking during subgraph expansion, prioritizing relations with stronger evidence and semantic consistency for inclusion in the record generation context. Thus, quality assessment affects the knowledge the model actually receives during generation, helping mitigate the propagation of uncertain relations through multi-hop combinations and subsequent generation.
This design controls how knowledge enters clinical record synthesis. Structural constraints address invalid connections, while reliability information distinguishes relations that pass these checks and prioritizes generation contexts with stronger evidence. For rare diseases, where knowledge is sparse and mechanistic relations across multiple levels must be combined, this helps reduce the risk of uncertainty in individual relations propagating into clinical records through knowledge composition.
Our clinical template is based on de-identified discharge summaries from MIMIC-IV-Note 2.2. We identify rare disease records using mappings from MIMIC-IV-EXT-RARE and summarize their common sections. We then merge synonymous section headings and standardize the content scope and ordering of the sections to form a standardized structure for clinical record synthesis.
The template retains the main elements of real clinical records related to the diagnostic and treatment process. Building on this structure, we explicitly add Differential Diagnosis as a separate section, requiring at least two alternative diagnoses with brief justifications. The record thus presents evidence supporting the target diagnosis and distinguishing it from similar diseases, in addition to the final disease name. For records with relevant mechanistic evidence, phenotypes, functional abnormalities, and genetic information can contribute to this organization, translating knowledge from pathogenic chains into evidence for clinical differential diagnosis and providing explanatory content for subsequent QA+CoT generation.
For Discharge Instructions, we further specify content relevant to rare diseases, including follow-up plans, prognosis, and, where applicable, genetic counseling and genetic testing recommendations. This addition connects genetic evidence and disease mechanisms with subsequent management, extending the record beyond disease identification.
Five experts reviewed the synthetic clinical records, and one expert performed the scoring. All experts were qualified physicians with at least three years of clinical experience