acceptodds
Under review as a conference paper at ICLR 2027

MedRDSyn: Knowledge Graph-Based Rare Disease Data Synthesis for Supervised Fine-Tuning

Abstract

To address the scarcity of real clinical records for rare diseases, we propose MedRDSyn, a knowledge-graph–guided framework for rare disease clinical note synthesis. MedRDSyn curates rare disease notes from MIMIC-IV-Note as clinical record templates and defines entity types and relation types required by clinical records. Through LLM-assisted incremental triple extraction from PubMed, MedRDSyn integrates Orphadata, HPO, and DrugBank, while using UMLS for entity normalization to construct a rare disease knowledge graph. The graph explicitly models associations among key medical entities, including genes, variants, proteins, diseases, and phenotypes, and contains 48,379 entities and 654,814 triples. Based on this knowledge graph, MedRDSyn performs LLM consistency verification with source-confidence fusion to construct a weighted graph, estimates node importance via weighted PageRank, and retrieves informative subgraphs through temperature-controlled seed sampling and beam-search expansion. After quality filtering, the framework generates 14,035 high-quality synthetic rare disease clinical cases and chain-of-thought data. Supervised fine-tuning on Qwen3-0.6B, Qwen3-4B, and Qwen3-8B using the synthesized data yields consistent improvements on PubMedQA, MedQA, ReDisQA, and RareBench, demonstrating the effectiveness of graph-guided rare disease data synthesis.

Primary area:
datasets and benchmarks
Keywords:
Rare disease, Knowledge graph, Data synthesis, Supervised Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YB5ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose MedRDSyn a knowledge-graph–guided framework for addressing the scarcity of real clinical records for rare diseases. The framework constructs a rare disease knowledge graph by integrating structured biomedical databases and unstructured text. It employs multiple-source triple generation, hierarchical multi-source knowledge fusion framework which includes entity alignment, ontology-based validation, and LLM-assisted relation weighting to improve the reliability of extracted knowledge. MedRDSyn then extracts relevant subgraphs to guide the generation of synthetic rare disease clinical cases. The resulting synthetic dataset, MedRDSyn-14k, is used for SFT of LLMs. The authors evaluate the approach on medical QA benchmarks.

Critical strengths
  • The authors introduce a framework for constructing a rare disease-oriented KG containing 48,379 entities and 654,814 triples.
  • The authors also introduce MedRDSyn-14k, a dataset containing 14,035 synthetic clinical samples that can be used to fine-tune medical LLMs, particularly for rare diseases where real clinical data is scarce. Unlike approaches that mainly rely on disease–phenotype associations, MedRDSyn incorporates pathogenic chains to guide the generation of more biologically plausible clinical records.
Critical weaknesses
  • A major concern is the substantial methodological and engineering complexity introduced by MedRDSyn relative to the reported performance gains. The framework requires multiple processing stages, including entity recognition, relation extraction, entity alignment, knowledge fusion, LLM-based relation weighting, PageRank computation, subgraph extraction and synthetic data generation etc… Several of these stages rely on additional LLM calls, potentially introducing considerable computational overhead. However, the reported performance gains appear relatively modest, with average improvements ranging from approximately +0.48 to +0.84 percentage points. Given the complexity of the proposed framework, the paper would benefit from a more thorough discussion of whether the performance improvements justify the additional computational and engineering costs.

  • Some individual improvements are particularly small, such as Qwen3-0.6B on MedQA with +0.12 or Qwen3-8B on PubMedQA with +0.20 points. Without statistical significance testing, it is difficult to determine whether these improvements are robust or fall within normal experimental variability.

  • The paper does not provide a sufficiently detailed analysis of the computational resources required to construct and use MedRDSyn. How much additional computational cost is introduced by: 1) Knowledge graph construction (+LLM usage) 2) Subgraph extraction and synthetic clinical case generation and 3) Additional supervised fine-tuning. A comparison of GPU-hours, inference costs, API usage, and total execution time would help assess the practical efficiency and scalability of the framework.

  • The authors state: "By constructing a rare disease-oriented KG and retrieving pathogenic-chain subgraphs, MedRDSyn explicitly supplements long-tail disease supervision signals and therefore continues to provide gains for larger models." However, the presented results do not directly demonstrate that the improvements are specifically attributable to better performance on rare or long-tail diseases. Indeed, the evaluation primarily only reports aggregate performance on general medical QA datasets. A more convincing evaluation would stratify benchmark questions according to disease prevalence or frequency in the training data, distinguishing between common and rare diseases. The authors could then investigate whether MedRDSyn provides significantly larger improvements for rare disease-related questions. Such an analysis would directly support the central motivation of the paper and demonstrate whether the proposed framework is particularly beneficial for long-tail medical knowledge.

  • The proposed knowledge graph is constructed through a largely automated pipeline. Although the authors introduce mechanisms intended to improve the reliability of extracted relationships, these mechanisms do not guarantee factual correctness. The paper would benefit from a quantitative human evaluation of a representative sample of graph triples. This is particularly important because errors in the KG may propagate into synthetic clinical records.

Questions for authors
  • In the related work section, the authors state: "Early KGs relied on expert construction (Smith et al., 2007), while recent automated pipelines construct disease-specific or rare disease KGs from structured resources and task-specific evidence graphs (Karki et al., 2025; Zhu et al., 2020; Qin et al., 2025). Nevertheless, existing automated rare disease KGs still face challenges such as data sparsity, relation noise, and constraint-based validation (Hogan et al., 2021)." Could the authors clarify how MedRDSyn specifically addresses the “relation noise and constraint-based validation” limitations?

  • How were the expert-validated clinical templates constructed? Could the authors clarify: How many experts participated, and what were their clinical qualifications? How were the experts recruited?

  • Could the authors justify their decision to use GPT-5.2 rather than specialized biomedical language models, such as BioBERT or PubMedBERT, for entity recognition and relation extraction? Previous studies, including “Benchmarking Large Language Models for Biomedical Natural Language Processing Applications and Recommendations” and “Exploring Transformer Models: Fine-Tuning vs. Inference on Relation Extraction from Biomedical Texts” report that specialized biomedical models can outperform general-purpose LLMs for biomedical relation extraction tasks.

  • Since the HMKF framework relies heavily on automated entity recognition, relation extraction, and LLM-based validation, errors may remain in the final graph. For example, some relations can be wrong either because entities are wrong or because the relation is false. Did the authors conduct a human evaluation of a representative sample of graph entities, relations, or pathogenic chains?

  • The authors primarily evaluate MedRDSyn through downstream medical question-answering benchmarks. Given that MedRDSyn is designed to address rare disease data scarcity, other downstream applications than QA could provide more direct evidence of its effectiveness. (e.g. Disease classification from clinical notes). Why is the downstream evaluation limited to medical Q/A?

5 comments

We thank the reviewer for the careful review. Below, we explain the design of knowledge validation and clinical record quality control, the construction of clinical templates, the choice of extraction model, and the evaluation settings for rare disease diagnosis.

  1. In the related work section, the authors state: "Early KGs relied on expert construction (Smith et al., 2007), while recent automated pipelines construct disease-specific or rare disease KGs from structured resources and task-specific evidence graphs (Karki et al., 2025; Zhu et al., 2020; Qin et al., 2025). Nevertheless, existing automated rare disease KGs still face challenges such as data sparsity, relation noise, and constraint-based validation (Hogan et al., 2021)." Could the authors clarify how MedRDSyn specifically addresses the “relation noise and constraint-based validation” limitations?

Our improvements address how knowledge from multiple sources is validated and used for clinical record synthesis. In an automatically constructed graph, a relation may satisfy type constraints while still lacking sufficient medical support. If such relations are treated equally during generation, noise retained during graph construction may propagate into clinical records and reasoning explanations.

We therefore apply quality control across knowledge fusion and generation. The schema, defined around clinical record requirements, enables relations from different sources to be checked against unified entity roles and connection rules. We then further assess the medical semantic reliability of relations that pass fusion and constraint checks. For each triple, the system retains the head and tail entities and constructs a positive statement from the original relation and a negative statement from its negation. GPT-OSS assesses the correctness of each statement and assigns a confidence score. By combining support for the positive statement with rejection of the negative statement, the system obtains a semantic consistency score for the relation, adding a reliability assessment beyond type constraints. This score is then combined with the source confidence obtained during fusion to produce the final relation weight. The weight contributes to node importance estimation and candidate relation ranking during subgraph expansion, prioritizing relations with stronger evidence and semantic consistency for inclusion in the record generation context. Thus, quality assessment affects the knowledge the model actually receives during generation, helping mitigate the propagation of uncertain relations through multi-hop combinations and subsequent generation.

This design controls how knowledge enters clinical record synthesis. Structural constraints address invalid connections, while reliability information distinguishes relations that pass these checks and prioritizes generation contexts with stronger evidence. For rare diseases, where knowledge is sparse and mechanistic relations across multiple levels must be combined, this helps reduce the risk of uncertainty in individual relations propagating into clinical records through knowledge composition.

  1. How were the expert-validated clinical templates constructed? Could the authors clarify: How many experts participated, and what were their clinical qualifications? How were the experts recruited?

Our clinical template is based on de-identified discharge summaries from MIMIC-IV-Note 2.2. We identify rare disease records using mappings from MIMIC-IV-EXT-RARE and summarize their common sections. We then merge synonymous section headings and standardize the content scope and ordering of the sections to form a standardized structure for clinical record synthesis.

The template retains the main elements of real clinical records related to the diagnostic and treatment process. Building on this structure, we explicitly add Differential Diagnosis as a separate section, requiring at least two alternative diagnoses with brief justifications. The record thus presents evidence supporting the target diagnosis and distinguishing it from similar diseases, in addition to the final disease name. For records with relevant mechanistic evidence, phenotypes, functional abnormalities, and genetic information can contribute to this organization, translating knowledge from pathogenic chains into evidence for clinical differential diagnosis and providing explanatory content for subsequent QA+CoT generation.

For Discharge Instructions, we further specify content relevant to rare diseases, including follow-up plans, prognosis, and, where applicable, genetic counseling and genetic testing recommendations. This addition connects genetic evidence and disease mechanisms with subsequent management, extending the record beyond disease identification.

Five experts reviewed the synthetic clinical records, and one expert performed the scoring. All experts were qualified physicians with at least three years of clinical experience

  1. Could the authors justify their decision to use GPT-5.2 rather than specialized biomedical language models, such as BioBERT or PubMedBERT, for entity recognition and relation extraction? Previous studies, including “Benchmarking Large Language Models for Biomedical Natural Language Processing Applications and Recommendations” and “Exploring Transformer Models: Fine-Tuning vs. Inference on Relation Extraction from Biomedical Texts” report that specialized biomedical models can outperform general-purpose LLMs for biomedical relation extraction tasks.

GPT-5.2 performs ontology-guided structured knowledge extraction to support rare disease clinical record synthesis, with entity recognition as one component. We first design the schema around the information needed for record generation. It covers patient information, symptoms and signs, diagnostic examinations, and therapeutic interventions in clinical records, while additionally incorporating genes, variants, proteins, phenotypes, and their connecting relations to organize rare disease pathogenic mechanisms. This yields unified definitions for 21 entity types and 28 relation types. GPT-5.2 extracts knowledge from unstructured literature under this schema. The model must interpret the literature according to the entity and relation definitions, assign mentions to the appropriate types, establish relations between entities, record relation attributes such as diagnostic findings, pathogenic mechanisms, and drug interaction endpoints, and retain the corresponding evidence from the source text. Together, these fields specify the medical meaning of a relation.

This task requires a variable number of entities, relations, and associated fields to be generated under unified ontology constraints, making it suitable for a generative language model with instruction-following capabilities. A decoder model can organize these interconnected fields into a unified structured output, retaining relations alongside their specific medical descriptions and evidence within the same extraction process. By comparison, encoder models such as BioBERT and PubMedBERT typically perform extraction through entity tagging or relation classification heads. Adapting them to our full output requirements would additionally require modules for attribute and evidence extraction.

These requirements make context preservation an important consideration in model selection. Relevant medical descriptions may span adjacent sentences or paragraphs, requiring context to resolve entity references, relations, and their applicable conditions. Both BioBERT [1] and PubMedBERT [2] are based on the BERT-BASE architecture. Their standard models use learned absolute positional embeddings and support input sequences of up to 512 tokens without positional encoding extension. GPT-5.2 provides a longer context window, allowing more room for ontology definitions, more complete literature passages, and structured output. Extraction results subsequently undergo entity normalization, relation validation, and multi-source evidence fusion. The retained relation attributes help distinguish and aggregate relations, while the source text evidence is provided with retrieved subgraphs to support subsequent clinical record and QA+CoT generation. The goal of this stage is therefore to provide structured knowledge with specific medical meaning and traceable literature evidence for record synthesis. As a general-purpose decoder model, GPT-5.2 is better suited to this task format.

[1] Lee J, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020;36(4):1234–1240. doi:10.1093/bioinformatics/btz682.

[2] Gu Y, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare. 2022;3(1):Article 2. doi:10.1145/3458754.

  1. Since the HMKF framework relies heavily on automated entity recognition, relation extraction, and LLM-based validation, errors may remain in the final graph. For example, some relations can be wrong either because entities are wrong or because the relation is false. Did the authors conduct a human evaluation of a representative sample of graph entities, relations, or pathogenic chains?

In addition to automated knowledge graph validation, we apply quality control to clinical records generated from combined graph knowledge before training. Rule-based filtering, overall quality assessment, and manual spot-checks are used to control the risk of potential errors entering the subsequent training data.

Rule-based filtering removes failed generations, empty texts, erroneous outputs, and records with abnormal lengths. Following Appendix C.3, GPT-OSS performs an overall quality assessment, focusing on content coherence, the clinical plausibility of the patient scenario, clearly unsafe or implausible descriptions, and the completeness of key clinical content. This step assesses the overall usability of graph knowledge once it has been combined into a specific patient scenario. Even when individual relations pass earlier checks, their integration into the medical history, examinations, diagnosis, and treatment course still requires assessment at the record level.

After automated filtering, we further review retained records through manual spot-checks. Appendix C.3 defines a score of 4 as good overall quality with minor issues, and a score of 5 as coherent content, a clinically plausible scenario, and complete key content. To allocate more expert review effort to samples that may still contain minor issues, we set sampling weights according to the automated scores: each record scored 4 receives three times the sampling weight of a record scored 5. This allows us to check for problems that automated assessment may have missed even in highly rated records.

Manual spot-checks use medical content validity as a single binary criterion, distinguishing pass from fail: 1 (pass) indicates medically accurate content with no obvious errors or severe medical implausibilities; 0 (fail) indicates clear medical errors or seriously implausible medical statements. The assessment focuses on whether the medical content meets basic validity requirements.

In this process, automated scoring filters out records with clear problems and prioritizes records for subsequent manual spot-checks. Manual spot-checks complement automated assessment by examining whether the retained records still contain obvious medical errors or seriously implausible content.

  1. The authors primarily evaluate MedRDSyn through downstream medical question-answering benchmarks. Given that MedRDSyn is designed to address rare disease data scarcity, other downstream applications than QA could provide more direct evidence of its effectiveness. (e.g. Disease classification from clinical notes). Why is the downstream evaluation limited to medical Q/A?

We include MedQA, PubMedQA, and ReDisQA to assess general medical knowledge and rare disease knowledge after training on the synthetic data. MedQA assesses clinical medical knowledge and its application to problem scenarios; PubMedQA assesses comprehension and judgment of evidence from biomedical literature; and ReDisQA assesses rare disease knowledge and reasoning. We also use RareBench to evaluate performance on disease prediction [3].

RareBench Task 4 evaluates phenotype-driven differential diagnosis of rare diseases. Given a patient's phenotypes or symptoms, the model generates and ranks the most likely candidate diseases. It must connect multiple patient findings with disease characteristics, distinguish between diseases that may have similar phenotypes, and assign priorities to the candidate diagnoses. The R@1 metric we report requires the model to identify the correct disease and rank it as the most likely diagnosis, reflecting both disease identification and the ability to rank the correct diagnosis first in differential diagnosis. RareBench thus evaluates prediction from patient clinical features to a disease diagnosis through open-ended generation, allowing us to examine the effect of synthetic training data on rare disease identification and diagnostic ability.

[3] Chen X, et al. RareBench: Can LLMs Serve as Rare Diseases Specialists? In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024:4850–4861. doi:10.1145/3637528.3671576.

Reviewer_8iVjOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MedRDSyn, a framework for synthetic rare disease clinical note generation. MedRDSyn first generates a knowledge graph from multiple resources with pathogenic chains. Then it uses a LLM to make this graph a weighted graph. It then uses GPT-OSS to generate synthetic rare disease clinical reports, and uses these reports to SFT Qwen3-XB models on 4 different benchmarks.

Critical strengths
  1. The paper addresses the scarcity of rare disease clinical training data by integrating clinical knowledge, especially the pathogenic chains into a generation pipeline. I personally find pathogenic chains very interesting. The knowledge graph and synthetic samples represents a good data construction effort.

  2. The paper is well-formatted and easy to follow.

Critical weaknesses
  1. The proposed generation pipeline combines existing techniques and methods, which limits the novelty of this paper. The pathogenic chain is one key contribution, but I doubt whether the pathogenic chain justifies the term "causal inference," as the framework primarily integrates biological mechanisms without explicitly performing causal inference. Another important thing is that the gains are relatively small between the MedRDSyn and other baselines (Table. 1); and among the ablation studies, which even do not have mutli-seed experiments (Table. 3). All together, these results do not provide enough evidence that the method complexity gives robust gains.

  2. The evaluation of synthetic generated clinical notes are mostly using downstream task's metrics rather than directly analyzing the clinical fidelity of the generated notes. Although the authors invite one expert and GPT-5.5 to do the quality assessment, the origin and evaluation of the rubric (Table. 7) is not clear, and the authors do not directly compare the synthetic data with real clinical notes, which should be an important evaluation process of synthetic datasets. Moreoever, it is not clear why the the number of 20,000 (inital) is chosen, and what is the cutoff for the exclusion (leaving 14,035 samples). Also, from Appendix D, the numbers of generated cases are different from different baseline methods, and the additional details (i.e. initial budget, filtering rules/rates, and disease coverage) of baseline methods are unclear, which makes it difficult to assess whether the comparisons are fully controlled and fair enough.

Questions for authors
  1. Could the authors clarify the "causal inference" in Figure. 2? Does the framework perform explicit causal inference?
  2. Could the authors provide multi-seed experiments and 95% confidence intervals for the main results and ablation studies?
  3. Could the authors compare the synthetic notes vs. real notes? i.e. on similarity, disease-level distribution, phenotype co-occurrence, etc.?
  4. Is the rubric in Table 7 validated/reviewed by clinical experts?
  5. What motivates the selection of 20,000 initial seed nodes? How does the downstream performance change with the number of synthetic training data (generated samples)? Could the authors provide a data scaling analysis to demonstrate the proposed method's advantages and consistent gains over other baseline methods?
  6. How are synthetic data generated for other baselines? Could the authors provide the results under matched number of training samples?
8 comments

We thank the reviewer for the specific suggestions concerning the presentation of our method and its experimental validation. We further explain the role of pathogenic mechanism knowledge in synthesis and provide additional repeated training runs, ablations, and comparisons across training data sizes. We address each question below.

  1. Could the authors clarify the "causal inference" in Figure. 2? Does the framework perform explicit causal inference?

The term “causal inference” is associated with the construction of pathogenic chains. In this work, pathogenic chains first serve to organize knowledge during knowledge graph construction. We treat genes, variants, protein function, clinical phenotypes, and diseases as interconnected knowledge levels. We integrate gene–phenotype and gene–disease associations from HPO and use variant information from UniProt to supplement variant–protein links, organizing genetic, molecular, and clinical knowledge scattered across different resources into a unified graph. This construction extends disease–phenotype associations upstream to genetic and molecular function levels, providing mechanistic knowledge to support rare disease clinical record synthesis.

These mechanistic relations enter the record generation context through retrieved subgraphs. When constructing patient scenarios, the model can jointly use genetic factors, molecular dysfunction, and associated clinical manifestations to organize record content around mechanistic links. Pathogenic chains thus enrich the medical facts available for synthesis and constrain consistency among genetic evidence, functional abnormalities, and phenotype descriptions in the generated records.

During subsequent QA+CoT generation, the model organizes its reasoning using the synthetic clinical record and the associated question and answer. Because the record already incorporates this mechanistic knowledge, the model can use relations linking genetic factors to functional changes and functional abnormalities to clinical phenotypes when explaining diagnoses or other clinical judgments. Through the sequence of knowledge graph, synthetic clinical record, and QA+CoT generation, pathogenic chains provide mechanistic evidence that guides the model to use these relations naturally in its explanations.

Therefore, “Causal inference” in Figure 2 refers to the use of existing pathogenic mechanism knowledge to guide the generation of clinical records and reasoning text, namely pathogenic mechanism-guided medical reasoning.

  1. Could the authors provide multi-seed experiments and 95% confidence intervals for the main results and ablation studies?

We repeat training of Qwen3-8B using data generated by MedRDSyn and RareSyn with three random seeds (42, 43, and 44), and evaluate the models on ReDisQA and RareBench. The results are presented below.

Table 1. Three-seed robustness check on rare-disease benchmarks using Qwen3-8B. Results are reported as mean ± standard deviation across three SFT seeds.

MethodReDisQA (%)RareBench (%)
RareSyn77.29 ± 0.1730.88 ± 1.17
MedRDSyn (ours)77.64 ± 0.2132.03 ± 0.25

MedRDSyn achieves higher mean performance than RareSyn on both rare disease benchmarks across repeated runs. Both methods exhibit low variation across seeds on ReDisQA. On RareBench, an open-ended diagnosis benchmark, MedRDSyn achieves a higher mean score with lower variation across seeds.

For the ablation study, we repeat training of the full method and three ablated variants on Qwen3-4B and Qwen3-8B with three training seeds (42, 43, and 44). The variants remove the knowledge graph context, pathogenic chains, and graph-aware retrieval, respectively. We evaluate them on MedQA, PubMedQA, and ReDisQA, as shown in Table 2.

Table 2. Multi-Seed Ablation Results with 95% Confidence Intervals.

ModelVariantMedQA (%)PubMedQA (%)ReDisQA (%)
Qwen3-4BMedRDSyn (ours)67.43 ± 0.6369.73 ± 0.5671.35 ± 0.50
w/o KG66.90 ± 0.2368.44 ± 0.4070.39 ± 0.78
w/o Chain67.30 ± 0.1869.00 ± 0.7470.90 ± 0.52
w/o Retrieval67.37 ± 0.3069.07 ± 0.5270.43 ± 0.60
Qwen3-8BMedRDSyn (ours)76.36 ± 0.7370.81 ± 0.2977.64 ± 0.52
w/o KG75.41 ± 0.6170.25 ± 0.2777.25 ± 0.57
w/o Chain75.01 ± 0.4370.07 ± 0.8477.36 ± 0.46
w/o Retrieval75.72 ± 0.5170.61 ± 0.2477.44 ± 0.58

Pathogenic chains connect genetic factors, molecular functional changes, and clinical phenotypes, introducing explanatory relations across these levels into the training content. Qwen3-8B is more sensitive to the removal of pathogenic chains, possibly because it can build on its existing medical knowledge to integrate these mechanistic links more fully into its interpretation of clinical records. The comparison between the two model sizes suggests that the mechanistic depth provided by pathogenic chains may be an important source of additional benefit for the larger model.

Random subgraph sampling still supplies graph facts, but the generation context no longer prioritizes relations according to their reliability, changing the combinations of evidence available for organizing clinical records. For Qwen3-8B, richer existing knowledge may reduce its reliance on this external organization, resulting in relatively smaller incremental benefits from retrieval on PubMedQA and ReDisQA. For Qwen3-4B, targeted evidence selection can provide clearer and more interconnected SFT supervision, reducing the burden of establishing knowledge connections from dispersed record content. Removing retrieval leads to a larger decrease in its ReDisQA performance.

  1. Could the authors compare the synthetic notes vs. real notes? i.e. on similarity, disease-level distribution, phenotype co-occurrence, etc.?

When constructing the template, we use disease mappings from MIMIC-IV-EXT-RARE to identify rare disease records in MIMIC-IV-Note. These records provide references for real clinical templates and constrain the format of the synthetic clinical records. To illustrate the connection between synthetic records and real clinical documentation, we provide a comparison of a real and a synthetic record documenting the same disease, drawn from the available data.

Both records document idiopathic intracranial hypertension (IIH) and share the combination of headache, diplopia, and papilledema. Both describe elevated cerebrospinal fluid opening pressure, no intracranial mass on imaging, empty sella findings, acetazolamide treatment, and weight management, showing similar disease manifestations and patterns of diagnosis and treatment. Table 3 presents the comparison.

Table 3. Clinical Comparison of Real and Synthetic Cases of Idiopathic Intracranial Hypertension.

Clinical dimensionMIMIC-IVMedRDSyn
Chief complaintHeadache for several weeks, with diplopia and tinnitus; history of obesity.Severe bifrontal headache, intermittent diplopia, and transient visual obscurations 10 days postpartum; headache began 3 days after delivery.
Examination findingsBilateral papilledema; CSF opening pressure of 55 cm H₂O with lymphocytic pleocytosis (WBC 35); MRI showing a partial empty sella without a mass or hydrocephalus; normal venous sinus flow on MRV.Bilateral optic disc edema and left sixth-nerve palsy; CSF opening pressure of 310 mm H₂O with normal CSF composition; MRI showing an empty sella without a mass or hydrocephalus; no cerebral venous sinus thrombosis on MRV.
DiagnosisIdiopathic intracranial hypertension (pseudotumor cerebri); aseptic meningitis, likely NSAID-induced.Idiopathic intracranial hypertension (postpartum).
TreatmentAcetazolamide 250 mg PO every 8 hours; acetaminophen as needed; weight-loss counseling and neuro-ophthalmology follow-up.Acetazolamide 500 mg PO twice daily; acetaminophen as needed; weight management and neurology/ophthalmology follow-up.

The synthetic record additionally presents the uncommon clinical scenario of IIH manifesting as postpartum headache with visual symptoms, which is not documented in the real record used for comparison. The supporting evidence can be traced to the subgraph retrieved during generation. This subgraph uses "postpartum headache" as its seed node, contains "IIH → HAS_SYMPTOM → postpartum headache", and also provides associations involving diplopia, optic disc edema, and acetazolamide treatment. A published clinical report likewise documents IIH as a rare cause of postpartum headache [1]. This example illustrates core disease features in the synthetic record that correspond to those in the real record, together with uncommon clinical associations supported by retrieval evidence.

[1] Mathew M, Salahuddin A, Mathew NR, Nandhagopal R. Idiopathic intracranial hypertension presenting as postpartum headache. Neurosciences (Riyadh). 2016 Jan;21(1):52-5. doi: 10.17712/nsj.2016.1.20150304.

Reviewer_SMNXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MedRDSyn, a knowledge graph-guided framework for synthesizing rare disease clinical data for supervised fine-tuning. The authors construct a rare disease knowledge graph by integrating multiple biomedical resources and introducing explicit pathogenic chains connecting genes, variants, proteins, phenotypes, and diseases. The framework further combines source confidence and LLM-based verification to estimate relation reliability, and uses weighted PageRank and beam search to retrieve informative subgraphs for clinical case generation.

The resulting knowledge graph contains 48,379 entities and 654,814 triples, from which 14,035 synthetic training samples are generated. The authors evaluate the synthesized data through supervised fine-tuning of Qwen3 models at three scales. Experiments on four medical QA benchmarks, together with clinical quality assessment and additional analyses, show consistent performance improvements.

Critical strengths
  1. The paper addresses an important challenge in rare disease research, particularly the limited availability of clinical training data. The integration of multiple biomedical knowledge sources and the explicit modeling of pathogenic chains are notable strengths. Unlike approaches that rely primarily on general medical knowledge or simple KG-based prompting, the proposed framework incorporates relation reliability and graph structure into the data synthesis process, providing a reasonable and clinically meaningful approach to generating rare disease training data.

  2. The authors evaluate three model sizes across four medical benchmarks and include ablation studies, clinical quality assessment, knowledge diversity analysis, and random-seed robustness experiments. These experiments examine both the generated data and the downstream models. Although the performance gains are generally modest, the consistent improvements and supporting analyses provide useful evidence for the effectiveness of the proposed framework.

Critical weaknesses
  1. Although the overall framework is well designed, several components rely on established techniques, including weighted PageRank, beam search, and heuristic confidence fusion. The current ablation studies demonstrate the value of KG context, pathogenic chains, and retrieval, but do not sufficiently isolate the contribution of each retrieval and weighting component. Furthermore, the pathogenic evidence usage analysis mainly measures relations appearing in generated rationales, which does not necessarily establish that the models have learned more reliable mechanistic reasoning. Additional evidence would strengthen the methodological contribution.

  2. The case generation process permits the LLM to infer or fabricate plausible clinical details when the retrieved subgraph lacks sufficient information. However, the medical correctness of these generated details is not independently verified at scale. The human evaluation includes only 30 cases per method and one clinical expert, and the human and LLM evaluators show a notable disagreement in consistency scores for MedRDSyn. Given that the downstream improvements are relatively modest, stronger quality validation would increase confidence in the reliability and practical value of the synthesized data.

Questions for authors

Q1. Could the authors provide statistics on the proportion of retrieved subgraphs that contain complete or meaningful pathogenic chains? It would also be useful to know how frequently the generated cases and QA+CoT samples explicitly use these relations. This would help clarify whether the proposed retrieval strategy consistently preserves the intended mechanistic evidence.

Q2. The framework allows the LLM to generate clinical details beyond those explicitly supported by the retrieved KG. How do the authors ensure that these additions are medically accurate and consistent with the available evidence? Could the authors provide more details about the validation criteria and quality-filtering thresholds?

Q3. Given the relatively small improvements on several benchmarks, could the authors provide additional statistical evidence for the main results, such as confidence intervals or repeated training runs? It would also be helpful to clarify whether the advantages remain when the training datasets are matched in size, and whether the generated dataset and KG will be released for reproducibility.

5 comments

We thank the reviewer for the careful review and constructive feedback. To address your questions about the use of mechanistic evidence, synthetic clinical record quality, and experimental reliability, we provide additional statistics, quality control details, and comparative results below.

Q1. Could the authors provide statistics on the proportion of retrieved subgraphs that contain complete or meaningful pathogenic chains? It would also be useful to know how frequently the generated cases and QA+CoT samples explicitly use these relations. This would help clarify whether the proposed retrieval strategy consistently preserves the intended mechanistic evidence.

To quantify the preservation of pathogenic mechanism evidence in retrieved subgraphs, we additionally measure the coverage of mechanistic relations along the main pathogenic chain. This measure is defined strictly in terms of relations between adjacent nodes in the Gene–Variant–Protein–Phenotype–Disease chain specified in our paper, namely Gene–Variant, Variant–Protein, Protein–Phenotype, and Phenotype–Disease. We check the entity types at both endpoints, the relation label, and the direction, requiring the corresponding directed edge to be present in the subgraph. Of the 20,000 retrieved subgraphs, 16,590 contain at least one mechanistic edge along the main chain that meets these criteria, corresponding to a coverage of 82.95%.

We introduce pathogenic chains to supplement and organize mechanistic knowledge across genes, variants, proteins, phenotypes, and diseases in rare diseases, providing interconnected medical evidence for clinical record synthesis. These relations are explicitly organized in the knowledge graph and retrieval context and serve as evidence for key medical facts during record generation, helping maintain consistency between genetic evidence and clinical manifestations. During subsequent QA+CoT generation, the model uses the synthetic clinical record to organize medical reasoning around the specific question. Knowledge from pathogenic chains thus provides indirect evidence through the record content. Depending on the question, reasoning may focus on diagnosis, differential diagnosis, diagnostic testing, or treatment, while remaining consistent with the medical facts in the record.

To examine the explicit use of medical relations in the final training samples, we follow the relation categories and identification criteria in Section 4.3. We randomly select 100 samples from the final MedRDSyn-14k dataset used for SFT and use GPT-5.5 to decompose each CoT into claims and assess their correctness. For claims judged correct, we determine whether they explicitly use medical relations among genes, proteins, phenotypes, diseases, tests, diagnoses, and treatments. Table 1 presents the results.

Table 1. Explicit use of medical relations in sampled training CoTs.

RelationDescriptionRelation Presence (%)
Gene–ProteinGenes, mutations, or genetic factors to changes in protein, enzyme, pathway, or cellular function40
Protein–PhenotypeMolecular dysfunction or pathophysiological abnormalities to symptoms, signs, or organ abnormalities61
Phenotype–DiseasePhenotypes or symptom combinations to disease identification99
Phenotype–DiagnosisSymptoms, signs, or family history to diagnostic support100
Test–DiagnosisLaboratory, imaging, pathology, or genetic testing findings to diagnostic support70
Diagnosis–TreatmentDisease or diagnosis to treatment, management, monitoring, or counseling39

The synthetic training CoTs show high coverage of routine clinical reasoning relations involving phenotype recognition, disease identification, and diagnostic support. They also include mechanistic links between genetic factors and molecular functional changes, and between pathophysiological abnormalities and clinical phenotypes. These findings indicate that the training text organizes clinical diagnostic evidence while incorporating explanations related to rare disease pathogenesis, providing supervision that connects genetic or molecular knowledge with patients' clinical manifestations.

Q2. The framework allows the LLM to generate clinical details beyond those explicitly supported by the retrieved KG. How do the authors ensure that these additions are medically accurate and consistent with the available evidence? Could the authors provide more details about the validation criteria and quality-filtering thresholds?

Content in synthetic clinical records can be divided into two categories:(1) Details that instantiate a synthetic patient. A knowledge graph typically does not specify an individual patient's exact age, symptom onset, test values, or hospitalization timeline, so the model must assign specific values to these attributes. For such additions, the generation instructions emphasize clinical plausibility and internal consistency. Test findings should fit the disease and clinical context, without clearly impossible values, the medical history, examinations, diagnosis, and treatment course should be mutually consistent, and the discharge diagnosis must agree with the provided evidence, as detailed in Appendix C.2. These additions focus on constructing a plausible and internally consistent patient scenario, with relatively less emphasis on adding medical knowledge.

(2) Assertions about medical knowledge itself. New disease–drug relations, inheritance patterns, or diagnostic interpretations of test results may change the medical meaning of the record and require support from the knowledge graph or independent medical evidence.

Accordingly, the details the model may add primarily concern patient simulation, instantiating existing medical knowledge in a plausible and internally consistent patient scenario. Statements concerning medical knowledge, including disease mechanisms, inheritance patterns, diagnostic evidence, and treatment relations, must be grounded in and consistent with the retrieved knowledge graph evidence.

During post-generation filtering, we remove failed generations, empty texts, and erroneous outputs. We then apply the overall quality assessment in Appendix C.3 and retain only records with a score ≥4. Experts subsequently perform weighted random spot-checks on the retained records. Each record scored 4 receives three times the sampling weight of a record scored 5, giving higher review priority to records that may still contain minor issues. Manual review uses a binary medical content validity criterion: 1 indicates medically accurate content with no obvious errors or severe medical implausibilities; 0 indicates clear medical errors or seriously implausible medical statements. This review checks the basic medical validity of records retained after automated filtering.

Q3. Given the relatively small improvements on several benchmarks, could the authors provide additional statistical evidence for the main results, such as confidence intervals or repeated training runs? It would also be helpful to clarify whether the advantages remain when the training datasets are matched in size, and whether the generated dataset and KG will be released for reproducibility.

We report repeated training runs and comparisons with matched training data sizes to further assess reliability.

We repeat training of Qwen3-8B using data generated by the two rare disease synthesis methods, MedRDSyn and RareSyn, with three training seeds (42, 43, and 44). We evaluate performance on ReDisQA and RareBench, as shown in Table 2.

Table 2. Three-seed robustness check on rare-disease benchmarks using Qwen3-8B. Results are reported as mean ± standard deviation across three SFT seeds.

MethodReDisQA (%)RareBench (%)
RareSyn77.29 ± 0.1730.88 ± 1.17
MedRDSyn (ours)77.64 ± 0.2132.03 ± 0.25

MedRDSyn achieves higher mean performance than RareSyn on both rare disease benchmarks across repeated runs. Both methods exhibit low variation across seeds on ReDisQA. On RareBench, an open-ended rare disease diagnosis benchmark, MedRDSyn achieves a higher mean score with lower variation across seeds, indicating greater stability in this open-ended diagnostic task.

We reproduce the synthesis and filtering procedures of each baseline's original method and use the resulting filtered datasets with sizes close to that of our method. To maintain consistency with the original procedures and fairness in comparison, we do not apply additional random sample selection or dataset truncation. The final training sizes therefore differ slightly: ClinGen, MedSyn, and RareSyn contain 13,709, 13,191, and 13,395 samples, respectively, compared with 14,035 for MedRDSyn. Each baseline differs from MedRDSyn by no more than 844 samples.

To further examine the effect of differences in training size, we truncate the synthesized and filtered datasets from all four methods to n = 13,000 and conduct comparisons with matched training data sizes on Qwen3-4B and Qwen3-8B. Table 3 presents the results.

Table 3. Results with Size-Matched Training Data

ModelMethodMedQA (%)PubMedQA (%)ReDisQA (%)
Qwen3-4BClinGen66.8569.2070.59
MedSyn67.0169.2070.88
RareSyn67.4869.0070.59
MedRDSyn67.7969.5071.25
Qwen3-8BClinGen75.8170.3077.06
MedSyn75.2670.1076.91
RareSyn76.0470.4077.43
MedRDSyn76.8370.7077.94

With training sample counts exactly matched, MedRDSyn outperforms all baselines on all three benchmarks at both model sizes. These results show that, under the evaluated settings, its advantages remain when training data sizes are matched, providing further support for the effectiveness of its synthetic data.

We plan to release the synthetic clinical records generated in this study to support reproducibility and further research.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.