acceptodds
Under review as a conference paper at ICLR 2027

Transformer-Based Multimodal Retrieval with Knowledge Graph Supervision for Clinical Decision Support

Abstract

Diagnostic decision support that combines imaging, unstructured clinical narra- tives, and structured tabular measurements is increasingly bottlenecked by threeproblems: (1) representations that ignore relational biomedical knowledge; (2) re-trieval mechanisms that are dominated by lexical or dense but semantically shallow matches; and (3) evidence sets that are redundant and unable to explain individual predictions. This paper introduces a knowledge-graph-supervised multimodal re-trieval transformer for diagnostic decision support whose primary methodological contribution is a coverage approximation guarantee. It jointly aligns imaging, textual, and tabular encoders under a unified biomedical ontology through a graph-aware supervision loss. The system retrieves candidate patient modal coverage, andfactual triples via a cross-modal attention transformer, and compresses the resulting candidate pool through a submodular utility, which jointly rewards diagnosticrelevance, cross-modal coverage, and ontological consistency while penalizing redundancy. Extensive experiments on MIMIC-IV coupled with MIMIC-CXR,TCGA multi-omic imaging cohorts, and ADNI multimodal Alzheimer data, usingthe Unified Medical Language System as the reference biomedical knowledge graph, show improvement over the strongest dataset-specific baseline of approximately 93.4% macro AUC and 87.6% macro F1, reducing average evidence-set size by 41.7% (24.7 to 14.4 facts), inference latency by 45.3% (342 to 187 ms),and expected calibration error by 47.5% (6.1% to 3.2%). Ablation studies and robustness tests under 30% modality dropout confirm that both the knowledge graph supervision loss and the submodular selection stage contribute independently and complementarily to these gains, positioning this model as a strong foundation for radiology triage and ward-level clinical decision support.

Primary area:
optimization
Keywords:
Biomedical informatics, cross-modal attention, diagnostic decision support, evidence set selection, knowledge graph supervision, multimodal fusio, retrieval augmented reasoning, submodular optimization.

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_kkrhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents KGMRT, a framework intended to combine imaging, clinical text, and tabular data for clinical decision support. The proposed approach uses UMLS-based supervision to align modality representations, a cross-modal transformer to retrieve candidate evidence, and a selection objective incorporating concept coverage, redundancy, and ontological consistency. The authors claim a 1−1/e1-1/e approximation guarantee for greedy evidence selection and report evaluations on several dataset, claiming improvements in classification performance, calibration, evidence-set size, and inference latency.

Critical strengths

None

Critical weaknesses
  1. Contradictory training descriptions and inconsistent experimental results. Section 3.5 states that Stage 2 jointly optimizes the classifier, encoders, and cross-modal transformer, whereas Appendix A states that the encoders and transformer are frozen after Stage 1. The reported results also conflict: Table 1 gives MIMIC-IV/CXR macro AUCs of 0.871 for AdaCoMed and 0.934 for KGMRT, while Table 3 reports 0.843 and 0.917 for the same methods without explaining a different evaluation setting. Furthermore, averaging the per-class values in Table 3 yields approximately 0.845 and 0.918, which are inconsistent with its own reported macro averages beyond ordinary rounding. Section 5.1 also names FusionNet as the strongest ADNI baseline, although it does not appear in Table 1. These discrepancies make both the training protocol and the reported performance unreliable as presented.

  2. Incomplete presentation and extensive citation errors. Appendix B contains an unresolved “Table ??” reference to TCGA results, but the corresponding table is missing. Table 3 is labeled as a MIMIC-CXR[1] evaluation while listing the NIH ChestX-ray14[2] disease categories, with no explanation of an alternative labeling procedure. The references for arXiv:2605.13277 and arXiv:2607.09982 contain incorrect authors and years, and several bibliography entries list “others” as the author. Multiple references also retain “UNVERIFIED” or “UNVERIFIED peer review” annotations. These unresolved verification markers, together with confirmed bibliographic errors, indicate that the references were not adequately checked before submission. Collectively, these problems substantially undermine the manuscript’s completeness and credibility.

[1] https://physionet.org/content/mimic-cxr-ext-mimic-cxr-db/1.0/

[2] Baltruschat IM, Nickisch H, Grass M, Knopp T, Saalbach A. Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classification. Sci Rep. 2019 Apr 23;9(1):6381. doi: 10.1038/s41598-019-42294-8. PMID: 31011155; PMCID: PMC6476887.

Questions for authors
  1. Stage 2 training: Section 3.5 states that the classifier, encoders, and cross-modal transformer are jointly optimized during Stage 2, whereas Appendix A states that the encoders and transformer are frozen. Which description matches the implementation used to obtain the reported results?
  2. Conflicting AUC results: Why do the MIMIC-IV/CXR macro AUCs in Table 1 (0.871 for AdaCoMed and 0.934 for KGMRT) differ from those in Table 3 (0.843 and 0.917)? Were different cohorts, splits, or evaluation protocols used?
  3. Within-table numerical inconsistencies: Averaging the per-class AUCs in Table 3 yields approximately 0.845 for AdaCoMed and 0.918 for KGMRT, rather than the reported macro averages. How were these averages calculated?
  4. Dataset labels: Why does Table 3 list the NIH ChestX-ray14 disease categories while identifying the benchmark as MIMIC-CXR? Were these labels independently constructed for MIMIC-CXR? If so, please explain the annotation procedure and provide the corresponding evaluation code.
  5. Missing results: Appendix B refers to TCGA per-cancer-type results through “Table ??”, but the table is absent. Were these experiments conducted, and can the authors provide the referenced results?
  6. Reference verification: How do the authors account for the incorrect authors and years associated with arXiv:2605.13277 and arXiv:2607.09982, the entries listing “others” as the author, and the retained “UNVERIFIED” annotations? Please verify the affected references and clarify whether the cited works support the associated claims.

I urge the authors to take responsibility for the accuracy and completeness of their manuscript and to verify it carefully before submission. Peer review should assess a rigorously prepared study, rather than serve as a chance-based opportunity for an unchecked manuscript to be accepted.

Reviewer_fgJaOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

KGMRT combines ontology-supervised multimodal encoders, cross-modal retrieval, and evidence selection. Results on MIMIC-IV/CXR, TCGA, and ADNI report improvements in classification, calibration, and efficiency. The stated main methodological contribution is a greedy coverage approximation guarantee.

Critical strengths
  1. The paper addresses a weakness of multimodal retrieval: individually relevant records can form a redundant or contradictory evidence set. Combining ontology-aware representation learning with set-level selection is a sensible response.
  2. The evaluation design considers evidence size, calibration, latency, missing modalities, and graph scale alongside classification. The component ablations also attempt to distinguish the roles of ontology supervision and evidence selection. These are useful evaluation targets, although the reported results need reconciliation.
Critical weaknesses
  1. Heavy reliance on an external concept linker without sensitivity analysis. The method relies heavily on the concept linker ϕ\phi to map patient data to UMLS. Any errors or omissions during the entity recognition stage will propagate as cascading errors, corrupting the graph supervision signal (LKG\mathcal{L}_{KG}) and the retrieval score. The paper lacks an analysis of how the system's performance degrades if a weaker extractor is used or if clinical texts contain non-standard abbreviations that cause linking failures.
  2. Ambiguous mathematical definitions and formatting errors. The key penalty term Inconsist(S, G) in Equation 6 is vaguely described in the text as counting contradictory relation triplets, entirely lacking a rigorous mathematical formula, which hinders reproducibility. Additionally, there are obvious formatting errors in the references, such as placeholder citations like "others, 2025a" and "others, 2026", and an unresolved cross-reference placeholder "Table ??" in Appendix B.
Questions for authors
  1. Please reconcile Tables 1 and 3 and their per-class arithmetic. Were the same models, label sets, and test patients used?
  2. How are patients and longitudinal records split, what note timestamps are permitted, how are labels derived, and which records can enter the retrieval index?
  3. Which Stage 2 training description is correct? Please specify the concept linker, concept-level modality representations, evidence item definition, and contradiction detector.
Reviewer_DQUpOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

KGMRT combines UMLS-supervised imaging, text, and tabular representations with cross-modal retrieval and evidence compression. Candidates are ranked using representation similarity and overlap between UMLS concept neighborhoods. A set objective then trades capped query-concept coverage against redundancy and ontological inconsistency, before the selected representations are pooled for classification. The paper evaluates MIMIC-IV/CXR, TCGA, and ADNI and presents a greedy approximation guarantee as a methodological contribution.

Critical strengths
  1. The paper addresses a useful set-level retrieval problem: highly ranked individual records need not form a compact, complementary evidence set. The separation between graph-aware representation supervision (Equations 3-4), candidate scoring (Equation 5), and evidence selection (Equation 6) makes the intended roles of the components explicit. Considering coverage and ontological conflicts alongside similarity is a sensible motivation for moving beyond score-ranked top-k retrieval.

  2. The evaluation considers more than diagnostic accuracy. Table 2 reports evidence-set size, calibration, and latency, including reductions from 24.7 to 14.4 evidence items and from 342 to 187 ms relative to AdaCoMed. Figure 4 and Section 5.3 ablate the graph loss, cross-modal attention, and selection stage; Figure 5 examines missing modalities. These are relevant tests of the proposed pipeline.

Critical weaknesses
  1. The approximation claim in Section 3.4 (p. 4) is not justified for Equation 6 as written. The capped coverage term is monotone, but subtracting the penalties need not preserve monotonicity. Under the natural interpretation that redundancy considers distinct pairs and is zero for a singleton, consider two candidates that cover the same query concepts, have positive similarity, and introduce no contradiction. Adding the second candidate leaves coverage unchanged while increasing the redundancy penalty, so F decreases. Appendix A uses a positive redundancy weight (mu = 0.4), and the paper states no condition excluding this case. The displayed maximum also does not explicitly exclude self-pairs; including them instead makes cosine self-similarity dominate the redundancy term for every nonempty set. The intended definition needs clarification. This monotonicity objection alone prevents applying the stated monotone-submodular greedy guarantee; submodularity of the exact maximum-similarity and inconsistency terms requires its own argument. This affects the paper's claimed theoretical contribution.

  2. The empirical result and training descriptions need reconciliation. Table 1 (p. 6) reports MIMIC-IV/CXR AUCs of 0.934 for KGMRT and 0.871 for AdaCoMed, whereas Table 3 (p. 11) reports macro averages of 0.917 and 0.843 on the 14-label MIMIC-CXR benchmark. If these are different evaluations, their cohorts, modalities, splits, and aggregation should be identified; the paper currently leaves their relationship unclear. Separately, Section 3.5 (p. 4) says the classifier is optimized jointly with the encoders and cross-modal transformer in Stage 2, while Appendix A (p. 10) says the encoders and transformer are frozen. Section 3.5 also describes selected-evidence pooling but calls selection an inference-time operation, leaving the classifier's training inputs unclear. These ambiguities obstruct reproduction and attribution of the reported improvements.

I recommend rejection because the central guarantee is unsupported under the stated objective and the experimental training recipe is internally inconsistent. A corrected objective or guarantee, together with a reconciled evaluation and training protocol, would be necessary to reassess the contribution.

Questions for authors
  1. What is the exact redundancy implementation, including self-pairs and singleton/empty-set values? What assumptions establish monotonicity and submodularity of the implemented objective, or what guarantee applies if it is non-monotone?

  2. How do Tables 1 and 3 relate? Please specify their evaluation cohorts, available modalities, splits, and macro-averaging protocols, and reconcile the reported AUCs.

  3. Which modules receive updates in Stage 2, and what evidence does the classifier receive during training? Figure 3 shows a 70/15/15 split: is this grouped by patient, and how are the retrieval gallery and repeated visits separated from validation/test queries?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.