acceptodds
Under review as a conference paper at ICLR 2027

Aligning Multimodal Patient Evidence with Biomedical Knowledge Graphs for Clinical LLMs

Abstract

Clinical questions often depend on linking a patient's multimodal evidence to external biomedical knowledge, yet existing predictive systems rarely represent such links explicitly, so they can neither be traced to their evidence sources nor removed to measure their contributions. We present MM-KG (**M**ulti**m**odal **K**nowledge **G**raph), which represents heterogeneous, multimodal patient observations and biomedical concepts as separate layers in one typed graph, joined by explicit alignment edges. First, modality-specific harmonizers convert EHR text, imaging, genomic, and biospecimen data into typed observations mapped to UMLS concepts, which a route-prioritized aligner links to a biomedical knowledge graph. Query-conditioned retrieval then selects a compact subgraph for downstream use by a large language model or a graph neural network. We build MM-KGs for MIMIC-IV and the Alzheimer's Disease Neuroimaging Initiative (ADNI) and evaluate them with a design that separates patient evidence, biomedical knowledge, and their interaction. On questions that require both sources, neither source alone performs far above chance, whereas their combination yields a drug-controlled AUROC interaction of on MIMIC and on ADNI. On held-out five-candidate ranking, MM-KG outperforms MindMap by Hits@1 and leads an adapted GraphCare on the items that require consulting the patient, and deleting the single answer-bearing relation from the retrieved packet returns Hits@1 to the no-knowledge baseline. Finally, query-conditioned retrieval reaches AUROC with less context than the strongest generic policy, whereas static knowledge graph context gives no consistent gain on ordinary outcome prediction. Knowledge graphs thus benefit clinical LLMs not as background context but as explicit links between multimodal patient evidence and the relation a question requires, and MM-KG makes these links retrievable, traceable, and testable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
knowledge graphs, clinical LLMs, multimodal healthcare

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UfkROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MM-KG, a framework that connects multimodal patient observations with biomedical knowledge graphs through explicit, provenance-preserving alignment edges. Patient observations and biomedical concepts are represented in separate graph layers, allowing their relationships to be retrieved and examined. The framework uses modality-specific harmonization, ontology alignment, and query-conditioned retrieval to provide relevant evidence to downstream LLMs and GNNs. The authors evaluate MM-KG on MIMIC-IV and ADNI using factorial experiments, baseline comparisons, and controlled removal of supporting knowledge. The results suggest that correctly linking patient evidence to biomedical relationships improves performance on questions that require both sources.

Critical strengths
  1. Explicit alignment between multimodal patient evidence and biomedical knowledge. I find the explicit separation of patient observations, biomedical concepts, and alignment edges to be the strongest aspect of this work. Unlike approaches that combine heterogeneous information into a single representation, MM-KG preserves the connections between observations and their supporting knowledge. This makes the retrieved evidence easier to trace and evaluate.

The correspondence-disruption experiments in Section 5.3 are particularly useful. Removing or modifying the relevant relationships substantially reduces performance, supporting the importance of correct patient–knowledge alignment.

  1. Careful experimental design and useful empirical findings. The factorial evaluation provides a meaningful way to examine the interaction between patient information and biomedical knowledge. The reported AUROC interactions of +0.194 on MIMIC and +0.299 on ADNI are encouraging. I also appreciate the retrieval-efficiency experiments and the observation that adding static knowledge graph context does not consistently improve ordinary clinical outcome prediction.
Critical weaknesses
  1. Limited contribution to representation learning. My main concern is the methodological novelty from a machine learning perspective. Although the framework includes a trained GNN for downstream prediction, its central components—graph construction, ontology alignment, and query-conditioned retrieval—are largely deterministic. The proposed representation is useful for organizing and tracing clinical evidence, but it is less clear whether MM-KG introduces a new learning mechanism or provides advantages beyond existing knowledge graph integration and retrieval techniques. A comparison with simpler symbolic patient–KG joins would help clarify this distinction.

  2. Benchmark construction and generalization remain concerns. The experimental design is generally strong, but the primary reasoning tasks are constructed around relationships already represented in PrimeKG. As a result, some improvements may reflect successful retrieval of known relationships rather than more general clinical reasoning.

I also note that MM-KG does not outperform the adapted GraphCare baseline on the full five-candidate ranking benchmark (0.575 versus 0.609 Hits@1), although it performs substantially better on the identity-hard subset (0.582 versus 0.375). These results are informative but suggest that the performance advantage depends on the evaluation setting.

Finally, the individual contributions of imaging, genomic, and other modalities are not fully established in the main reasoning experiments. Stronger evidence of multimodal benefits would make the contribution more convincing.

Questions for authors
  1. Beyond the downstream GNN, what aspects of MM-KG are learned? Can the authors clarify the methodological contribution relative to deterministic knowledge graph construction and retrieval?

  2. How does MM-KG compare with a simpler approach that directly joins normalized patient observations with relevant PrimeKG relations, without constructing the proposed multimodal graph?

  3. How well does MM-KG generalize to clinical questions defined independently of PrimeKG, particularly questions involving unseen or incomplete biomedical relationships?

  4. Can the authors provide clearer evidence of the individual contributions of imaging, genomic, and other modalities to the main reasoning tasks?

3 comments
AuthorsOpenReview ↗

Response to Reviewer UfkR

“Beyond the downstream GNN, what aspects of MM-KG are learned? Can the authors clarify the methodological contribution relative to deterministic knowledge graph construction and retrieval?”

MM-KG’s contribution lies in its representation rather than a new learning mechanism: it keeps multimodal observations, their provenance, and their alignment to the knowledge graph separately addressable, now explicit in the abstract and contribution statements. Three components are learned (now stated in Sec. 4): the CheXpert-calibrated imaging thresholds, the normative brain-MRI model, and the GNN. Alignment and retrieval are deterministic by design, so retrieved relations are label-free, traceable, and reproducible, and any source, modality, or edge can be removed with all else fixed. These choices still matter empirically: retrieval policies at the same token budget range from 0.526 to 0.731 AUROC (App. Table 15), and replacing normalized concepts with raw report text costs 0.042 AUROC (App. Q.3).

“How does MM-KG compare with a simpler approach that directly joins normalized patient observations with relevant PrimeKG relations [...]”

We ran exactly this (App. Q.1, Table 22): a fixed rule joining normalized observations to the candidate drug's PrimeKG relations, with no graph or LLM. Adding one normalization step at a time, it reaches AUROC 0.545 (~chance) from coded diagnoses alone (the input of code-based methods such as GraphCare), 0.892 with report entity linking, and 0.987 with our value-aware laboratory rules. With full normalization it essentially reproduces the labels, which are defined by this match, so we now report it as an upper bound on achievable performance (Sec. 3.5). It is accurate only because it uses MM-KG's normalization and alignment, and it provides no per-observation tracing, per-edge ablations, or LLM and GNN consumers.

“How well does MM-KG generalize to clinical questions defined independently of PrimeKG, particularly questions involving unseen or incomplete biomedical relationships?”

We have scoped our claims accordingly (Sec. 5.1, conclusion): our benchmarks measure whether a model connects patient evidence to a given relation, not clinical reasoning beyond PrimeKG. Results reflect use of the supplied relation rather than memorized knowledge: the interaction is the same for contraindications that real-world prescribing corroborates (+0.145) and contradicts (+0.150) (App. G). Relabeling the contraindication items from FDA drug labels, independently of PrimeKG (App. Q.7), only the KG condition recovers label-defined contraindications (label-balanced accuracy 0.632 vs. 0.508 closed-book). Where the label lists a contraindication PrimeKG lacks, the model recovers 31% from parametric knowledge; where both are silent, the KG condition increases false positives (24% vs. 4%). The benefit is thus bounded by PrimeKG's coverage, as Sec. 5.1 now states.

“I also note that MM-KG does not outperform the adapted GraphCare baseline on the full five-candidate ranking benchmark [...]”

The advantage depends on whether candidate-drug identity alone predicts the label. GraphCare is supervised on 19,085 labeled pairs (MM-KG is zero-shot), and its score rises when its patient input is removed (Table 3), so it partly ranks drugs by identity. On a rebuilt benchmark where identity is uninformative (App. Q.6), MM-KG reaches 0.583 Hits@1 vs. 0.458 for retrained GraphCare and 0.444 for MindMap. On the original split, combining both methods' scores reaches 0.720, above either, suggesting complementary information (Table 3, Sec. 5.2).

“Can the authors provide clearer evidence of the individual contributions of imaging, genomic, and other modalities to the main reasoning tasks?”

We collected all per-modality results in a new table (App. Table 31). On the patient–KG questions, laboratory values and report findings have the largest effect (removing them costs 0.148 and 0.123 Hits@1), while image-derived findings are redundant with the report. With the report withheld (App. Q.5), image-derived findings help only together with the KG (interaction +0.146 AUROC). Imaging and biomarkers also contribute to the GNN (−0.008 AUROC without imaging) and to ADNI prognosis (+0.057 balanced accuracy), while targeted genomic relations add nothing. The abstract now states which evidence channel carries which result.

AuthorsOpenReview ↗

Gentle reminder: looking forward to your feedback on our response

Dear Reviewer UfkR,

Thank you again for your thoughtful review. We would like to check whether our response resolves your concerns. In particular, we compared MM-KG with a direct symbolic patient–KG join, evaluated on FDA-label contraindications defined independently of PrimeKG, and rebuilt the ranking benchmark so that drug identity alone does not predict the label; there, MM-KG outperforms retrained GraphCare (0.583 vs. 0.458 Hits@1). We look forward to any further feedback and would appreciate you considering an updated score if your concerns are resolved.

Best regards, The authors

AuthorsOpenReview ↗

Final note before the discussion closes

Dear Reviewer UfkR,

Thank you again for your review. As the discussion period closes today, please let us know if any concerns remain that we could clarify; we would be happy to respond.

Best regards, The authors

Reviewer_8V3WOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work stands within the larger literature area of combining patient longitudinal histories with biomedical concepts and knowledge graphs. While this area is not new, existing work uses techniques like RAG to retrieve relevant concepts and enrich base patient history in text or in graph form.

In contrast, the authors here propose a method to make explicit edges (called alignment edges) from the patient's multimodal history related nodes to known KG concepts. This avoids collapsing the patient's history and KG inputs just once into one fused, richer history. It instead makes a materialized graph connecting each encounter/observation to KG concepts for multiple downstream queries. This formulation supports query conditioned retrieval, allowing different subgraphs from this graph to be retrieved on the fly as queries change. This is the key idea in their design which also has potential for other usecases in this community beyond point of care queries (as discussed below).

In addition to the motivation and treatment of the problem, the paper also performs extensive experimentation in Section 5. These experiments examine the interaction value of patient evidence and biomedical KG inputs, compare and prod existing methods like GraphCare, evaluate retrieval accuracy along with token efficiency and perform ablations. The adversarial experiments to destroy answer bearing KG edges and shuffle packets is also designed rigorously for deep analysis of the differential value of their method.

Overall, the framing and the design of their evaluation setup clarify the approach's usefulness well.

Critical strengths

Overall, this work could be useful to the community for the following key reasons:

  1. Patient evidence and external biomedical concepts in KGs are kept separately but linked with explicit alignment edges in a materialized graph at an observation to concept level. This retains encounter and source provenance granularly. Beyond this, it makes query specific retrieval and evidence tracing possible as the authors discuss comprehensively. Additionally, beyond their discussed use-case of point-of-care clinical care like contraindication checks, I see potential for this patient+KG graph to be used for other use-cases such as cohorting of patients for RWE (real world evidence) research or clinical trial matching as well. Based on the use-case, the KG and alignment edges can be leveraged differently or used for symbolic constraint based search too. As a result, beyond a monolithic patient+KG reasoning system, this observation level multimodal concept linking can facilitate various other use cases.

  2. The experimental design and analysis is rigorous. The 2x2 design in Section 5.1 validates the main hypothesis of the value of patient and KG interaction. The GraphCare experiment and identity-hard evaluation set adds nuance to how GraphCare leads at first glance, while MMKG has a differentiated advantage on the identity-hard setup (where a drug candidate based prior fails). The adversarial setup in 5.3 involving the removal of the answer bearing KG relation and shuffling packets also does well to show that merely adding any graph text doesn't add the same value that correct observation level KG linking does to the final results. Retrieval and ablation sections are adequately rigorous as well. The negative result for KG context in ordinary mortality prediction is also valuable, since it keeps the value of biomedical knowledge realistic rather than suggesting that adding a KG universally improves prediction.

Critical weaknesses

The key critique is that the authors have not shown how query-conditioned retrieval actually changes the alignment edges in the retrieved subgraph with changes in queries for the same patient. The main advantage of their architectural novelty is in this dynamic retrieval possibility instead of a one-time fused KG enrichment baseline. It would help to have a heatmap of query-to-alignment-edge usage and overlap statistics across retrieval subgraphs. Additionally, a manual clinician annotated evaluation set for assessing the alignment edge quality and the relevance of the retrieved evidence to each query could further strengthen their proposal.

Similarly, since this graph preserves observation level evidence with encounter and timestamp provenance, I was expecting to see atleast a case study (if not statistical analysis) of temporal reasoning. Particularly, it would be useful to see how a longitudinal graph can support point-of-care reasoning when queries are based on historical conditions vs. current conditions of the patient too.

Questions for authors

Materially impactful:

  1. When you remove the answer-bearing KG edge - are these only for edges fully contained in the KG? Or do you also destroy critical alignment edges? The latter could be useful to see if the alignment edges are critical for retrieving the right subgraph and the impact on the final reasoning prompt. It would be useful to see if the downstream reasoning LLM can infer the correspondence from the remaining patient and KG evidence, if retrieval errs on the side of comprehensiveness?

Suggestions for future work: 2. Have you considered using the materialized graph for tasks beyond point of care clinical reasoning? Using the same patient in getting to RWE research cohorts or clinical trial matching could highlight additional advantages of your architecture, particularly if different query intents use different patient observations, alignment edges and biomedical KG relations. 3. Consider adding query-conditioned retrieval on the patient's history as well not just KG subgraph retrieval. Patient histories in some situations (like lengthy oncology histories) could benefit from query conditioned retrieval to reduce noise in the final reasoning context.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8V3W

“The key critique is that the authors have not shown how query-conditioned retrieval actually changes the alignment edges in the retrieved subgraph with changes in queries for the same patient. [...] Additionally, a manual clinician annotated evaluation set [...] could further strengthen their proposal.”

Thank you for both suggestions. In our experiments, only the KG side of retrieval is query-conditioned; the patient block is the same for every query about an admission, as Sec. 3.4 now states more explicitly. We computed the requested heatmap and overlap statistics (App. Q.2, Figs. 7–8). Across question families for the same admission, the alignment edges on a correspondence path change with each family's relation whitelist (median Jaccard overlap 0); across the five candidate drugs of the ranking benchmark, only the correct candidate uses any edges, by construction of the decoys. Under the patient-side retrieval you suggest (see below), the retrieved patient subgraph itself changes with the query (Fig. 7b).

Clinician annotation was unfortunately not feasible within the discussion period; we now list it as a limitation (App. R) and consider it for future work. As automated proxies, our report-derived findings agree with the CheXpert and NegBio reference labelers at F1 0.65–0.90 (App. Q.8), and in our ADNI citation audit all 85 cited relations were in the supplied packet (App. L).

“[...] I was expecting to see atleast a case study (if not statistical analysis) of temporal reasoning. [...]”

Thank you for the suggestion; we added one such case study in App. Q.9, Fig. 9. There, two patients receive magnesium sulfate (contraindicated for kidney disease in PrimeKG): one has a renal condition coded only in a prior admission, the other a high creatinine in the current stay. Their subgraphs differ in source and timestamp, and for both the model scores the temporally matching query (“history of” vs. “currently” contraindicated) higher; rendering the stored source and time in the prompt widens this margin.

“When you remove the answer-bearing KG edge - are these only for edges fully contained in the KG? Or do you also destroy critical alignment edges? [...]”

Only PrimeKG relations: with a query-independent patient block, the alignment edge is not a prompt token, so removing it would not change the LLM's input (now stated in Sec. 5.3). Under patient-side retrieval through the edges (see below), deleting the single answer-bearing alignment edge removes only that observation, and Hits@1 falls from 0.686 to 0.297, the no-knowledge level (new row in Table 4).

On whether the LLM can infer the correspondence when retrieval is comprehensive: partly. Condition D supplies the full admission, yet in 327 of 1,199 positive items the needed relation is in the prompt and the answer is still incorrect; once the answer-bearing observation is removed, the rest of the record does not let the model recover the match.

“Have you considered using the materialized graph for tasks beyond point of care clinical reasoning? [...]”

We agree; the 29.9% statistic in our introduction is such a query: a cohort defined as a symbolic constraint over patient observations, alignment edges, and PrimeKG relations. Eligibility criteria such as “condition X in any modality, on drug class Y, without Z” take the same form, and each qualifying patient carries the evidence that qualified them. We added this to App. R and the Conclusion.

“Consider adding query-conditioned retrieval on the patient's history as well not just KG subgraph retrieval. [...]”

We implemented this as an additional analysis (Sec. 5.4, App. Q.3): aligned observations are kept only if their concept appears in the candidate drug’s PrimeKG relations, the rest of the admission is kept as context, and the rule was fixed before test scoring. At the same token count, AUROC rises from 0.731 to 0.806 and Hits@1 from 0.575 to 0.686 (same direction for two other LLMs), and, consistent with your intuition about noise, unused-relation failures fall from 327 to 300. Because this rule relies on the same patient–relation match that defines the labels, part of the gain reflects the retriever itself performing the match; long histories with independently defined labels, as in your oncology example, are a natural next step.

AuthorsOpenReview ↗

Gentle reminder: looking forward to your feedback on our response

Dear Reviewer 8V3W,

Thank you again for your detailed and constructive review. Following your suggestions, we added the query-to-alignment-edge heatmap and overlap statistics, a temporal case study, and patient-side query-conditioned retrieval, where removing the single answer-bearing alignment edge drops Hits@1 to the no-knowledge level. We would be glad to hear whether these address your concerns, and we are happy to clarify anything further.

Best regards, The authors

Reviewer_8V3WOpenReview ↗

Response to improvements

Thank you for the additional experiments. It is interesting to see the case study and the patient history experiments that improve the AUROC from 0.731 to 0.806. Also, the experiment where you remove the critical alignment edges impacting the Hits@1 to 0.297 is also useful to my original question. Thank you for the comprehensive response, hope this makes your work better and more extensible to other use-cases as discussed above.

Changed my overall grade to: 4.

Many thanks for increasing your score! We appreciate the helpful feedback to improve our paper, and are happy to hear that the new results are useful.

Best, The authors

Reviewer_Eo1tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MM-KG: it places multimodal patient evidence and PrimeKG into the same typed graph, connects “patient observations—biomedical concepts—relations required by the question” with traceable and removable alignment edges, and then retrieves compact subgraphs on demand to pass to an LLM or GNN. Its core claim is that the value of knowledge graphs for clinical models lies not in generic background knowledge but in explicitly recovering the correspondences between patient states and external relations. It has three main contributions: an instance–knowledge graph construction framework that is separately addressable and ablatable, query-conditioned retrieval, and a 2×2 factorial evaluation that separates patient evidence, KG relations, and their interaction. In validation, the drug-controlled AUROC interaction on MIMIC is +0.194, and on the ADNI test set it is +0.299 (p=0.0135); five-candidate Hits@1 is 0.575 (random 0.2), removing the answer-bearing alignment edges reduces it to 0.299, and shuffling the evidence pack reduces it to 0.185; retrieval outperforms the strongest general strategy with 0.731 AUROC and 1,206 tokens versus 0.712 AUROC and 8,157 tokens. Overall, the experimental design and mechanistic validation are relatively solid, but the method looks more like a systematic integration of existing KG-RAG, graph retrieval, and GNN components.

Critical strengths

None

Critical weaknesses
  1. The methodological innovation is mainly systems integration. Entity linking, graph retrieval, RAG, provenance, and GNN consumption are all existing components. MM-KG’s novelty lies in separating the patient instance layer from the knowledge graph layer, per-observation provenance, removable alignment edges, and the 2×2 factorial evaluation; these are system design and evaluation methods, not new retrieval or reasoning algorithms.

  2. “Not background context, but explicit links” is not truly isolated. What the LLM ultimately sees is still serialized text, not graph topology. Deleting answer-bearing edges and shuffling evidence packs can show that “correct relational content matters,” but cannot show that “the graph structure itself is more useful than ordinary KG-RAG context with equivalent facts.”

  3. A genuinely matched standard KG-RAG baseline is missing.

Questions for authors
  1. The methodological innovation is mainly systems integration. Entity linking, graph retrieval, RAG, provenance, and GNN consumption are all existing components. MM-KG’s novelty lies in separating the patient instance layer from the knowledge graph layer, per-observation provenance, removable alignment edges, and the 2×2 factorial evaluation; these are system design and evaluation methods, not new retrieval or reasoning algorithms.

Q1.1: Apart from “per-observation provenance” and “removable alignment edges,” what irreplaceable capability does MM-KG add compared with the closest prior systems such as MedGraphRAG, GraphCare, and MindMap? Please demonstrate it experimentally, not merely with the capability checklist in Table 1.

Q1.2: Is the paper’s main contribution ultimately a new method or an ablatable evaluation framework? If it is a method, please clearly specify the new algorithm or learning objective; if it is an evaluation framework, we recommend adjusting the title, abstract, and contribution statements accordingly, so as to avoid giving readers the impression that it proposes a new reasoning paradigm independent of KG-RAG.

  1. “Not background context, but explicit links” is not truly isolated. What the LLM ultimately sees is still serialized text, not graph topology. Deleting answer-bearing edges and shuffling evidence packs can show that “correct relational content matters,” but cannot show that “the graph structure itself is more useful than ordinary KG-RAG context with equivalent facts.”

Q2.1: The paper’s abstract says that the knowledge graph is “not as background context but as explicit links,” yet §3.5 explicitly states that the LLM receives serialized text of Gp,q\mathcal{G}_{p,q}. Please provide an operational definition of “explicit link” and explain, given that the LLM only sees serialized statements/triples, what essential difference there is between this and the context in ordinary KG-RAG.

  1. A genuinely matched standard KG-RAG baseline is missing.

Q3.1: Please add a strictly matched conventional KG-RAG baseline: it uses the same patient records, the same candidate knowledge graph packs, the same serialization method, and the same token budget, but does not include MM-KG’s explicit alignment edges or per-observation provenance. Please report the differences between this baseline and MM-KG in AUROC, Hits@1, and token cost.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Eo1t

“Q1.1: Apart from “per-observation provenance” and “removable alignment edges,” what irreplaceable capability does MM-KG add compared with the closest prior systems such as MedGraphRAG, GraphCare, and MindMap? [...]”

MM-KG builds on established components, but adds a capability these systems lack: deriving biomedical concepts from measured values and images, each linked to its concept by a removable edge. For example, none of them can obtain “renal impairment” from a creatinine of 4.3 mg/dL, as this never appears in the record as text. We added two experiments to show this. With no LLM (App. Q.1), a fixed rule joining the patient's observations to the candidate drug's PrimeKG relations gives AUROC 0.545 (~chance) from coded diagnoses alone, 0.892 when adding report entity linking (as in text-based pipelines), and 0.987 when adding our value-aware laboratory rules (+0.40 on the laboratory-based questions). With the same LLM (App. Q.4), a text-linker KG-RAG pipeline reaches 0.660 AUROC and 0.353 Hits@1 vs. 0.731 and 0.575 for MM-KG, as it retrieves the needed relation for only 35% of positives (vs. 99.7%). Related Work now describes this capability.

“Q1.2: Is the paper’s main contribution ultimately a new method or an ablatable evaluation framework? [...]”

As stated in our general comment above in more detail, the contribution is in the representation and the evaluation it enables, not a new algorithm or reasoning paradigm, and we have revised the abstract, introduction, and contribution statements to reflect this. We maintained the title, as it describes the paper's task without claiming a new reasoning paradigm.

“Q2.1: [...] Please provide an operational definition of “explicit link” and explain [...] what essential difference there is between this and the context in ordinary KG-RAG.”

We agree and apologize for the confusing wording; we did not intend a claim about graph structure. The abstract and conclusion now state the claim our experiments test: KGs help when retrieval supplies the specific relation that matches the patient’s observed condition. An explicit link is an ALIGNS-TO edge from one patient observation (e.g., a creatinine value) to at most one PrimeKG concept (e.g., kidney disease), frozen before evaluation. It is never a prompt token; instead, it defines the benchmark labels, makes it measurable whether any retrieval policy's context contains the needed relation, underlies the evidence certificates, and is a GNN edge (Sec. 3.4). The difference from ordinary KG-RAG is what is retrieved: the queried drug's relations of the type the question needs, rather than context about entities in the record. At the same budget, generic KG-RAG contains the needed relation for at most 26% of positives (AUROC 0.526–0.638) vs. 99.7% for MM-KG (0.731; App. Table 15). When patient observations are instead retrieved through their alignment edges (App. Q.3), deleting the one answer-bearing edge lowers Hits@1 from 0.686 to 0.297, the no-knowledge level.

“Q3.1: Please add a strictly matched conventional KG-RAG baseline [...] but does not include MM-KG’s explicit alignment edges or per-observation provenance. [...]”

Sec. 5.4 now reports these baselines together. Since the alignment edge is not a prompt token, the baseline as specified is our condition D with only the provenance headers removed. It retrieves the same relations and patient evidence, so it is MM-KG with a slightly different prompt rendering rather than a competing method; the effect is small and opposite across cohorts (ADNI AUROC −0.011, MIMIC Hits@1 +0.027), and the headers serve auditability rather than accuracy.

Isolating what conventional KG-RAG lacks requires baselines that select their own relations from the KG (App. Table 15). First, standard KG-RAG policies (SapBERT vector retrieval, full admissible context, typed one-hop expansion) capped at MM-KG’s budget reach 0.526–0.638 AUROC vs. 0.731. Second, uncapped, the strongest (vector retrieval) reaches 0.712 but uses 8,157 tokens vs. 1,206. Third, a new text-linker KG-RAG baseline (App. Q.4) with the same LLM, KG, and token cap reaches 0.660 AUROC and 0.353 Hits@1 vs. 0.731 and 0.575 (p < 0.001).

AuthorsOpenReview ↗

Gentle reminder: looking forward to your feedback on our response

Dear Reviewer Eo1t,

Thank you again for your review! We wanted to follow up while the discussion period is still open and see if we addressed your concerns. In particular, for your main question of what MM-KG adds over conventional KG-RAG, we added a matched text-linker KG-RAG baseline, which obtained 0.660 AUROC vs. 0.731 for MM-KG, with the same LLM, KG, and token budget, and an analysis of which normalization steps make the patient–knowledge match possible. We would welcome any further discussion, and if these address your concerns, we hope you would consider revisiting your score.

Best regards, The authors

AuthorsOpenReview ↗

Final note before the discussion closes

Dear Reviewer Eo1t,

As today is the last day of the discussion period, we wanted to send one final note. We believe our response and revised paper address the points raised in your review, in particular the comparison with KG-RAG and the framing of our contribution, and we would be glad to answer any remaining questions today. Thank you again for your time.

Best regards, The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of responses to reviewer feedback

Thank you to all reviewers for their helpful suggestions and valuable feedback. We address each comment point-by-point below, and describe the corresponding changes to the paper. Notably, we have added new experiments, with results added to the main text (Sec. 5 and Tables 3–4) and full details in Appendix Q, including a matched text-linker KG-RAG baseline (Q.4), a deterministic join that isolates each modality harmonizer (Q.1), patient-side retrieval through the alignment edges and alignment-edge removal (Q.3), query-to-alignment-edge analyses (Q.2), a ranking benchmark where candidate-drug identity is uninformative (Q.6), an external check against FDA-label contraindications (Q.7), imaging when the report is withheld (Q.5), alignment audits (Q.8), and a temporal case study (Q.9), as well as a limitations and extensions section (App. R). The main text also now clarifies that MM-KG’s contribution is a representation that keeps each multimodal observation, its provenance, and its link to the knowledge graph individually addressable, as well as the factorial evaluation that this separability enables. The text also now explains what the alignment edges contribute even though the LLM only receives serialized text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.