- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MM-KG, a framework that connects multimodal patient observations with biomedical knowledge graphs through explicit, provenance-preserving alignment edges. Patient observations and biomedical concepts are represented in separate graph layers, allowing their relationships to be retrieved and examined. The framework uses modality-specific harmonization, ontology alignment, and query-conditioned retrieval to provide relevant evidence to downstream LLMs and GNNs. The authors evaluate MM-KG on MIMIC-IV and ADNI using factorial experiments, baseline comparisons, and controlled removal of supporting knowledge. The results suggest that correctly linking patient evidence to biomedical relationships improves performance on questions that require both sources.
Critical strengths
- Explicit alignment between multimodal patient evidence and biomedical knowledge. I find the explicit separation of patient observations, biomedical concepts, and alignment edges to be the strongest aspect of this work. Unlike approaches that combine heterogeneous information into a single representation, MM-KG preserves the connections between observations and their supporting knowledge. This makes the retrieved evidence easier to trace and evaluate.
The correspondence-disruption experiments in Section 5.3 are particularly useful. Removing or modifying the relevant relationships substantially reduces performance, supporting the importance of correct patient–knowledge alignment.
- Careful experimental design and useful empirical findings. The factorial evaluation provides a meaningful way to examine the interaction between patient information and biomedical knowledge. The reported AUROC interactions of +0.194 on MIMIC and +0.299 on ADNI are encouraging. I also appreciate the retrieval-efficiency experiments and the observation that adding static knowledge graph context does not consistently improve ordinary clinical outcome prediction.
Critical weaknesses
-
Limited contribution to representation learning. My main concern is the methodological novelty from a machine learning perspective. Although the framework includes a trained GNN for downstream prediction, its central components—graph construction, ontology alignment, and query-conditioned retrieval—are largely deterministic. The proposed representation is useful for organizing and tracing clinical evidence, but it is less clear whether MM-KG introduces a new learning mechanism or provides advantages beyond existing knowledge graph integration and retrieval techniques. A comparison with simpler symbolic patient–KG joins would help clarify this distinction.
-
Benchmark construction and generalization remain concerns. The experimental design is generally strong, but the primary reasoning tasks are constructed around relationships already represented in PrimeKG. As a result, some improvements may reflect successful retrieval of known relationships rather than more general clinical reasoning.
I also note that MM-KG does not outperform the adapted GraphCare baseline on the full five-candidate ranking benchmark (0.575 versus 0.609 Hits@1), although it performs substantially better on the identity-hard subset (0.582 versus 0.375). These results are informative but suggest that the performance advantage depends on the evaluation setting.
Finally, the individual contributions of imaging, genomic, and other modalities are not fully established in the main reasoning experiments. Stronger evidence of multimodal benefits would make the contribution more convincing.
Questions for authors
-
Beyond the downstream GNN, what aspects of MM-KG are learned? Can the authors clarify the methodological contribution relative to deterministic knowledge graph construction and retrieval?
-
How does MM-KG compare with a simpler approach that directly joins normalized patient observations with relevant PrimeKG relations, without constructing the proposed multimodal graph?
-
How well does MM-KG generalize to clinical questions defined independently of PrimeKG, particularly questions involving unseen or incomplete biomedical relationships?
-
Can the authors provide clearer evidence of the individual contributions of imaging, genomic, and other modalities to the main reasoning tasks?
3 comments
Gentle reminder: looking forward to your feedback on our response
Dear Reviewer UfkR,
Thank you again for your thoughtful review. We would like to check whether our response resolves your concerns. In particular, we compared MM-KG with a direct symbolic patient–KG join, evaluated on FDA-label contraindications defined independently of PrimeKG, and rebuilt the ranking benchmark so that drug identity alone does not predict the label; there, MM-KG outperforms retrained GraphCare (0.583 vs. 0.458 Hits@1). We look forward to any further feedback and would appreciate you considering an updated score if your concerns are resolved.
Best regards, The authors
Final note before the discussion closes
Dear Reviewer UfkR,
Thank you again for your review. As the discussion period closes today, please let us know if any concerns remain that we could clarify; we would be happy to respond.
Best regards, The authors
Response to Reviewer UfkR
“Beyond the downstream GNN, what aspects of MM-KG are learned? Can the authors clarify the methodological contribution relative to deterministic knowledge graph construction and retrieval?”
MM-KG’s contribution lies in its representation rather than a new learning mechanism: it keeps multimodal observations, their provenance, and their alignment to the knowledge graph separately addressable, now explicit in the abstract and contribution statements. Three components are learned (now stated in Sec. 4): the CheXpert-calibrated imaging thresholds, the normative brain-MRI model, and the GNN. Alignment and retrieval are deterministic by design, so retrieved relations are label-free, traceable, and reproducible, and any source, modality, or edge can be removed with all else fixed. These choices still matter empirically: retrieval policies at the same token budget range from 0.526 to 0.731 AUROC (App. Table 15), and replacing normalized concepts with raw report text costs 0.042 AUROC (App. Q.3).
“How does MM-KG compare with a simpler approach that directly joins normalized patient observations with relevant PrimeKG relations [...]”
We ran exactly this (App. Q.1, Table 22): a fixed rule joining normalized observations to the candidate drug's PrimeKG relations, with no graph or LLM. Adding one normalization step at a time, it reaches AUROC 0.545 (~chance) from coded diagnoses alone (the input of code-based methods such as GraphCare), 0.892 with report entity linking, and 0.987 with our value-aware laboratory rules. With full normalization it essentially reproduces the labels, which are defined by this match, so we now report it as an upper bound on achievable performance (Sec. 3.5). It is accurate only because it uses MM-KG's normalization and alignment, and it provides no per-observation tracing, per-edge ablations, or LLM and GNN consumers.
“How well does MM-KG generalize to clinical questions defined independently of PrimeKG, particularly questions involving unseen or incomplete biomedical relationships?”
We have scoped our claims accordingly (Sec. 5.1, conclusion): our benchmarks measure whether a model connects patient evidence to a given relation, not clinical reasoning beyond PrimeKG. Results reflect use of the supplied relation rather than memorized knowledge: the interaction is the same for contraindications that real-world prescribing corroborates (+0.145) and contradicts (+0.150) (App. G). Relabeling the contraindication items from FDA drug labels, independently of PrimeKG (App. Q.7), only the KG condition recovers label-defined contraindications (label-balanced accuracy 0.632 vs. 0.508 closed-book). Where the label lists a contraindication PrimeKG lacks, the model recovers 31% from parametric knowledge; where both are silent, the KG condition increases false positives (24% vs. 4%). The benefit is thus bounded by PrimeKG's coverage, as Sec. 5.1 now states.
“I also note that MM-KG does not outperform the adapted GraphCare baseline on the full five-candidate ranking benchmark [...]”
The advantage depends on whether candidate-drug identity alone predicts the label. GraphCare is supervised on 19,085 labeled pairs (MM-KG is zero-shot), and its score rises when its patient input is removed (Table 3), so it partly ranks drugs by identity. On a rebuilt benchmark where identity is uninformative (App. Q.6), MM-KG reaches 0.583 Hits@1 vs. 0.458 for retrained GraphCare and 0.444 for MindMap. On the original split, combining both methods' scores reaches 0.720, above either, suggesting complementary information (Table 3, Sec. 5.2).
“Can the authors provide clearer evidence of the individual contributions of imaging, genomic, and other modalities to the main reasoning tasks?”
We collected all per-modality results in a new table (App. Table 31). On the patient–KG questions, laboratory values and report findings have the largest effect (removing them costs 0.148 and 0.123 Hits@1), while image-derived findings are redundant with the report. With the report withheld (App. Q.5), image-derived findings help only together with the KG (interaction +0.146 AUROC). Imaging and biomarkers also contribute to the GNN (−0.008 AUROC without imaging) and to ADNI prognosis (+0.057 balanced accuracy), while targeted genomic relations add nothing. The abstract now states which evidence channel carries which result.