- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents RareDx, a framework for rare disease diagnosis, which includes RareDx-Harness and RareDx-KGPO. RareDx presents a method to normalize heterogeneous records into one ranked diagnosis task while RareDx-KGPO is a reinforcement learning method that projects predictions into a canonical disease graph with ontology information. The authors evaluate RareDx across 8 benchmarks, and RareDx achieves nearly the best or second best resutls against several baselines.
Critical strengths
-
The evaluates RareDx across 8 rare disease benchmark datasets with heterogeneous patient representations. This broad evaluation is a useful and strong evidence for the proposed RareDx method.
-
The integration of ontology information into the RareDx-KGPO is motivated for rare disease diagnosis since many related diseases can share the overlapping phenotypes. Although the individual components in KGPO are not exactly new, their combination provides a reasonable and domain valid approch for integrating graph information into reinforcement learning.
Critical weaknesses
-
The methodology novelty of RareDx-Harness is not clear. Most of the components in RareDx-Harness (Figure. 1) are exist. It is not clear what new methodology contributions (i.e. decision-making) the RareDx-Harness itself provides. More importantly, the definition of 'Harness' seems to be unclear between the RareDx-Harness methodology and the result section about the model description. In Figure. 1, the 'Harness' presents a framework consisting of Direct, Static RAG, Adaptive ReAct, and Structured 3-hop. However, in the result section (Table. 1), the models named RareDx-XB, Harness seems to only choose between Direct and a 35B Diagnostic Audit. Therefore, it is hard to determine what the system is actually responsible for, and their relationship is not clear. It is also hard to determine how the components Static RAG, Adaptive ReAct, and Structured 3-hop contribute to the full system.
-
The experimental evaluation does not sufficiently isolate the contributions of the proposed methods. Since the 35B audit model might be used by the router, the performance gains cannot be attributed solely to the 9B or 27B model. Furthermore, the reward components are not ablated for the downstream diagnostic performance, making their contribution not clear in this case. These limitations weaken the evidence supporting the paper's claims, and it is difficult to assess whether the components are responsible for the observed improvements.
Questions for authors
- How do strategies in Figure. 1 contribute to the final system? Can the authors further clarify what methodology contribution the proposed Harness method provides beyond selecting between Direct and Audit? Additionally, could the authors apply the same 35B Diagnostic Audit / Harness to other baseline models to see if the gains are consistent?
- Since the operational router selects between Direct and Audit using development data, how would the system determine which way to do when encoutering a new cohort without label development data? Considering this is an application paper, could the authors provide evidence of generatlization to unseen settings?
- How are evaluation inputs preprocessed to ensure that the ground truth diagnosis related information is excluded in the dataset, particularly in the free-text records and cases containing the genetic findings? Could the authors also report the unique diseases, diseases frequency, and accepted diagnoses per case for each dataset? This will help to improve the transparency of the paper.
- Given the relatively low performance of RareDx, especially Hit@1, what is the intended real world clinical usage of RareDx? How could it help the clinicians to do the diagnosis?
- Could the authors provide ablations on the reward components in KGPO?
- Could the authors explain the inconsistency about the title and abstract of the submitted pdf file and those displayed on OpenReview? For example, the OpenReview abstract refers to RareDx-28B, whereas the PDF reports RareDx-27B. The descriptions of the methodology and experimental results also differ in some details.
5 comments
Thank the authors for the additional tables and clarifications. These additional results address my concerns about the Harness definition, dataset statistics, and evaluation.
My two major concerns are still not addressed.
- As mentioned in my Weakness 2, the contributions of Harness and KGPO are still not fully isolated. An important missing ablation is Base Qwen3.5-9B + Harness (35B Audit) without KGPO. Although the authors acknowledged that this factorial ablation is missing, I still consider it to be an important experiment for attributing the final-system gains.
- Could the author extend Table 5 to provide downstream ablations (i.e. Hit@1,5,10) for individual KGPO reward components?
A smaller concern: while the authors state that the gold labels are loaded separately, have you checked whether the evaluation inputs (i.e. free-text or genetic findings) can directly reveal the diagnoses? I have checked RareArena’s paper (Chen et al., 2026), and the authors stated that they perform potential diagnosis leakage screening (not even remove the labels, but change some wording accordingly) during the dataset construction. Is a similar screening step ensured for other benchmarks? Have the authors done the dataset screening and cleaning in advance?
These remaining concerns are the main reasons I am maintaining my current rating 2.
We thank the reviewer for the constructive follow-up and for confirming that our previous response resolved the questions on the Harness definition, dataset statistics, and evaluation setup. Full results (Tables 1-5) are at https://anonymous.4open.science/r/RAREDX_rebuttal/rebuttal4.md.
1. Factorial Ablation of Base Model, KGPO, and Harness
We evaluated Base Qwen3.5-9B + Harness (35B Audit) without KGPO (as well as Top-10 SFT-9B and unrouted Always-Audit) under both the Full 8-source protocol and the Strict 80% disjoint test split (20% dev-selected route). Complete breakdowns and 27B results are in Tables 1-2; key rows are:
| System | Protocol | H@1 | H@5 | H@10 |
|---|---|---|---|---|
| Base 9B + Direct | Full / Strict test | 9.53 / 9.29 | 20.69 / 20.16 | 24.77 / 24.34 |
| Base 9B + Harness (35B Audit) | Full / Strict test | 11.83 / 11.30 | 25.09 / 24.10 | 30.00 / 28.83 |
| Top-10 SFT-9B + Harness | Full / Strict test | 16.08 / 15.12 | 27.73 / 26.62 | 34.41 / 33.26 |
| KGPO 9B + Direct | Full / Strict test | 16.90 / 16.29 | 25.54 / 24.71 | 31.35 / 30.54 |
| KGPO 9B + Harness (35B Audit) | Full / Strict test | 20.11 / 18.81 | 31.54 / 30.21 | 38.34 / 37.34 |
- KGPO alone outperforms Base 9B + Harness without 35B Audit calls: Under both protocols, KGPO 9B + Direct surpasses Base 9B + Harness at every cutoff (+5.07 H@1 and +1.35 H@10 on Full; +4.99 H@1 and +1.71 H@10 on Strict test).
- Complementary roles: The 35B Audit has no retrieval or tools and only reranks the policy's Top-10 list under a 384-token cap. When Base 9B omits the true disease neighborhood (e.g., Phenopackets 20.6%, LIRICAL 19.7%; Table 2), Audit cannot recover it. KGPO populates the initial Top-10 list with ontology- and phenotype-relevant candidates, amplifying the Harness gain (+5.23 H@10 on Base 9B vs. +6.99 on KGPO 9B).
2. Downstream KGPO Reward Ablations
We extended our previous Table 5 by training matched 9B policies from the same SFT checkpoint, ablating each reward channel () and safeguard (). Full results (including Harness scores, nDCG , and per-dataset scores) are in Tables 3-4; key Direct results are:
| Reward Configuration (Trained from SFT-9B) | Val Top-1 | Direct H@1 | Direct H@5 | Direct H@10 | OOV (%) |
|---|---|---|---|---|---|
| Stage 1: Top-10 SFT Initialization | 78.22 | 13.43 | 22.03 | 27.59 | 4.18 |
| Full RareDx-KGPO | 85.09 | 16.90 | 25.54 | 31.35 | 0.62 |
| w/o Graded-Gold Relevance () | 79.51 | 13.84 | 23.13 | 28.90 | 0.85 |
| w/o Ontology Proximity () | 82.62 | 15.46 | 23.64 | 29.14 | 0.74 |
| w/o Phenotype Profile () | 82.19 | 15.68 | 23.75 | 29.28 | 0.71 |
| w/o Biomedical Embedding () | 84.12 | 16.31 | 24.95 | 30.61 | 0.68 |
| Sparse Exact-Match Only () | 80.15 | 14.53 | 22.89 | 28.24 | 0.92 |
| w/o Vocab Gate & Contamination () | 74.46 | 12.14 | 19.84 | 24.13 | 28.45 |
| w/o Output-Budget Gate () | 81.33 | 15.13 | 23.45 | 28.73 | 1.84 |
Removing drops H@1 by 3.06 points because dense channels are capped (). Removing or reduces H@10 by 2.21 and 2.07 points (-3.11 for Sparse Exact-Match Only). Removing the vocabulary gate () raises fabricated/OOV strings from 0.62% to 28.45% and collapses H@10 below SFT (24.13% vs. 27.59%).
3. Input Modality and Leakage Screening
- 0% free-text narrative in the six non-RareArena benchmarks: RareArena (RDS/RDC) uses published clinical case reports, requiring narrative wording screening (Chen et al., 2026a). In contrast, MyGene2, RAMEDIS, MME, HMS, LIRICAL, and Phenopackets (2,368 cases) contain 0.00% free-text clinical narratives (Table 5); inputs are strictly structured HPO term lists (e.g., HP:0001943) plus candidate gene symbols, precluding narrative wording leakage by construction.
- Programmatic screening across all 4,249 prompts: Scanning all serialized test prompts against the 27,554-disease vocabulary and accepted synonym sets (exact substring and token-span Levenshtein distance ) confirmed 0 / 2,368 (0.00%) matches in the six structured sets and 0.00% explicit gold-label presence in RDS/RDC (Table 5).
- Genetic findings: Genotype tracks supply candidate gene symbols rather than disease names, mapping to 3.9 to 5.2 distinct rare diseases per gene on average in our knowledge graph (pleiotropy; Table 5), which requires joint phenotype-genotype disambiguation (Base 9B reaches only 7.2%-10.2% H@1 on gene-containing sets).
We will incorporate Tables 1-5 into the revision. We would be grateful if you would kindly consider raising your score.
We thank the reviewer for the detailed questions and for recognizing our broad evaluation and motivated ontology-grounded RL. Full results (Tables 1-6) are at https://anonymous.4open.science/r/RAREDX_rebuttal/rebuttal3.md.
W1 & Q1: Harness definition and final-system contribution
We agree that “Harness” denoted two objects. RareDx-Harness is an evaluation methodology, not a novel routing algorithm. It fixes the task, Top-10 contract, knowledge layer, normalizer, and metrics while varying evidence access. Figure 1's Direct, Static RAG, ReAct, and Structured 3-hop are controlled studies of when knowledge helps; they do not all feed the final prediction.
Operationally, a dataset-level router chooses frozen Direct or Diagnostic Audit. The 35B auditor sees the case and Direct list with a fixed prompt, greedy decoding, 384-token cap, and no retrieval/tools. We will rename the two objects and revise Figure 1 (Table 3).
We have not audited every baseline, so cannot claim model-invariant audit gains. Proprietary baselines had no audit; we withdraw any implication that the 9B policy alone exceeds GPT-5.5. The attribution is:
Thus policy training and compute-augmented auditing provide distinct gains; only the first two rows are a policy-only comparison. Table 4 gives every development decision and both frozen test candidates.
Q2: a new cohort without labeled development data
The router is not a case-level gate. For an unlabeled cohort, the prespecified default is Direct. Audit is enabled only if representative labeled local development data support it under the fixed Hit@10, Hit@5, Hit@1 rule. Eight source-held-out evaluations show breadth, not prospective cross-hospital validation; we will not claim zero-shot router generalization.
Q3: label isolation, overlap, and dataset composition
Loaders use only designated free text, phenotypes, and benchmark-supplied genetic findings. Gold diseases are loaded separately by the evaluator and, during training, remain in reward metadata rather than the prompt. Genetic findings are intended evidence, not labels.
Against all test sources we checked normalized identifiers, SHA-1 case text, and disease-HPO profiles at Jaccard at least 0.90. SFT/RL training have zero identifier and zero text matches. SFT has zero profile matches; RL has one with LIRICAL among 61,918 prompts. RL validation has 290 profile matches with Phenopackets but no identifier/text match, potentially favoring selection. Excluding Phenopackets leaves Harness Hit@10 at 37.47 versus GPT-5.5's 35.41. Retrieval contains disease-level ontology/gene records, not benchmark patient reports.
Table 1 reports cases, unique primary diagnoses, median/maximum disease frequency, and accepted strings per case. RDS/RDC have a median of six accepted strings because aliases are accepted for one primary diagnosis, not because each case has six independent gold diseases.
Q4: intended clinical use
RareDx is a differential-diagnosis support tool, not an autonomous diagnostician. Its Top-10 list helps prioritize phenotype review, gene interpretation, and confirmatory testing. Hit@1 measures first choice; Hit@5/10 measure screening retention. Modest Hit@1 is a limitation: predictions require expert verification and must not determine treatment. We will foreground the lack of prospective validation.
W2 & Q5: reward-component attribution
The audit tests reward behavior, not downstream causality. A phenotype neighbor scores 0.316 versus 0.008 without graph/phenotype credit. Without validity safeguards, a pseudo-disease changes from -0.175 to +0.320 and is rewarded in 82.5% of probes; placing gold at rank 11 changes 0.279 to 0.068 (Table 5). On 932 unseen cases, exact Top-1 rises from SFT's 78.22% to KGPO's 85.09%. This supports complete KGPO, not each term; matched train-from-SFT ablations are required, and we will narrow the claim.
Q6: OpenReview/PDF inconsistency
This is a version-control error, not a second model. “28B” was an earlier rounded label; the evaluated checkpoint is Qwen3.8-27B. We will make the OpenReview title/abstract match the PDF verbatim, use RareDx-27B throughout, and record the checkpoint identifier (Table 6).
We appreciate these important concerns. We will separate research from deployment, disclose overlap, and narrow unsupported claims. We would be grateful if you would kindly consider raising the score. Thank you again for your time and constructive feedback.