- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes GraphWalker, a method for selecting historical patient cases as demonstrations for a frozen LLM. It organizes training patients into a graph, retrieves initial cases from relevant cohorts, and then selects demonstrations using a context-dependent information-gain proxy. Candidate scores are updated as cases are added, and graph expansion allows the search to move beyond the initial retrieval set. The evaluation covers mortality, length-of-stay, and readmission prediction, with additional experiments on transfer, different backbones, and medical question answering.
Critical strengths
The empirical improvements are substantial in several important settings. For example, with Qwen3-14B, mortality AUPRC increases from 32.23 for the strongest baseline to 59.46, while AUROC increases from 75.37 to 84.19 (Table 1). Results across different tasks and backbones give the method a reasonable empirical foundation. The appendix strengthens the case that the selection procedure contributes beyond the patient representation. Table 8 gives competing selectors access to the same SMART-based retrieval and still shows a substantial advantage for GraphWalker on mortality. Table 11 separately tests rescoring and graph expansion while fixing both the initial candidates and the final demonstration count. These are useful controls, and they provide more convincing support for the method than the main comparison alone.
Critical weaknesses
I could not fully reconstruct the clinical prediction timeline from Appendix B.1–B.2. The input consists of visits preceding the target visit, but the relationship between the “last visit” used as the target and the subsequent ICU readmission window is unclear. A valid index-stay construction may be intended, but the distinction between hospital admissions and ICU stays needs to be explicit. The observation window and variable alignment used for TJH also need more detail. These choices determine what information is available at prediction time and how the transfer results should be interpreted. Some of the smaller advantages need more cautious interpretation. For example, LLaMA readmission AUPRC is 46.37 versus 46.32 for EHR-emb, and the matched global top-9 control ties GraphWalker on readmission AUROC. Reporting means over three seeds does not show the precision of these comparisons. The large improvements remain meaningful, but the evidence for the smaller incremental benefits is less clear.
Questions for authors
- What exactly constitutes a visit for each task, and which stay is the readmission index? Please give a timeline showing the observed history, excluded target records, and outcome follow-up window.
- For TJH, which measurements are available at prediction time, and how are its temporal observations and variables aligned with the MIMIC-IV representation?
- Can you provide paired confidence intervals for the principal comparisons, especially against the SMART-enhanced selectors and the matched global top-9 control?
2 comments
Response to Reviewer WLLW (2/2)
W2 & Q3: Paired Confidence Intervals for the Principal Comparisons
We thank the reviewer for this suggestion and agree that means over three runs do not show the precision of the comparisons. We therefore report paired bootstrap confidence intervals for the comparisons highlighted by the reviewer. Consistent with the reviewer's assessment, the intervals support the larger gains, including the advantage over the strongest SMART-enhanced selector on all three metrics and the mortality AUPRC gain from cohort initialization, whereas the smaller differences remain uncertain.
Experimental setup. For each comparison, we resample test patients with replacement. In each resample, both methods are evaluated on the same patients; each metric is computed for each of the three runs and averaged over runs, consistent with the reported means, and we take the difference between GraphWalker and the comparator. We report the difference on the full test set together with the 2.5th and 97.5th percentiles of the resampled differences. F1 uses the same fixed decision threshold as in the main results.
Table R2. Paired Bootstrap Differences (GraphWalker − Comparator)
Differences in percentage points, with pointwise 95% confidence intervals in brackets; positive values favor GraphWalker. Bold: the interval lies entirely above zero.
| Setting | Comparator | AUROC | AUPRC | F1 |
|---|---|---|---|---|
| MIMIC-III Mortality, LLaMA-3.1-8B (Table 8) | Delta-KNN + SMART | +9.40 [5.42, 13.67] | +6.97 [1.31, 13.24] | +6.07 [1.58, 10.20] |
| MIMIC-III Mortality, Qwen3-14B (Table 10) | w/o Cohort (global top-9)† | +2.03 [−0.67, 4.62] | +8.44 [3.65, 13.12] | +1.69 [−2.63, 6.00] |
| MIMIC-IV Readmission, Qwen3-14B (Table 10) | w/o Cohort (global top-9)† | 0.00 [−1.65, 1.78] | +0.84 [−1.63, 3.40] | +1.00 [−1.33, 3.44] |
| MIMIC-IV Readmission, LLaMA-3.1-8B (Table 1) | EHR-emb | +4.57 [0.89, 8.30] | +0.05 [−4.81, 4.86] | +1.73 [−1.61, 5.11] |
† The Table 10 comparator uses global top-9 initialization followed by the same graph expansion and early-stopping rules as GraphWalker.
Analysis & Conclusion.
First, the advantage over the strongest SMART-enhanced selector is supported across all three metrics. On MIMIC-III Mortality with LLaMA-3.1-8B, GraphWalker improves AUROC / AUPRC / F1 over Delta-KNN + SMART by 9.40 / 6.97 / 6.07 points, and all three confidence intervals lie entirely above zero. The remaining SMART-enhanced selectors trail GraphWalker by larger margins on all three metrics (Table 8). Since the comparator also uses SMART-based retrieval, this provides further support for the contribution of GraphWalker's selection procedure beyond access to an EHR-specific encoder.
Second, cohort initialization provides a clear mortality AUPRC gain with the initial candidate count matched. Against global top-9 initialization with Qwen3-14B, GraphWalker improves AUPRC by 8.44 points [3.65, 13.12]. Both methods start from nine candidates and retain the same subsequent search procedure, supporting the value of cohort-based entry points for mortality prediction. The AUROC and F1 intervals include zero, so the evidence for this incremental benefit is strongest on AUPRC.
Third, readmission results distinguish a supported AUROC gain from uncertain smaller differences. With LLaMA-3.1-8B, GraphWalker improves AUROC over EHR-emb by 4.57 points [0.89, 8.30]. The near-tied AUPRC difference of 0.05 points [−4.81, 4.86] and the F1 interval do not establish additional gains. Likewise, all three intervals for Qwen3-14B against global top-9 initialization include zero. We therefore interpret these smaller margins with caution, while the larger gains above remain supported.
We thank the reviewer again for the constructive suggestions, which helped us make the prediction protocol explicit and the interpretation of our results more precise. We hope these responses address the reviewer's concerns.
References
[1] Zhu et al. ClinicRealm: Re-evaluating large language models with conventional machine learning for non-generative clinical prediction tasks. npj Digital Medicine, 2026.
[2] Johnson et al. MIMIC-III, a freely accessible critical care database. Scientific Data, 2016.
[3] Johnson et al. MIMIC-IV, a freely accessible electronic health record dataset. Scientific Data, 2023.
[4] Yan et al. An interpretable mortality prediction model for COVID-19 patients. Nature Machine Intelligence, 2020.
[5] Gao et al. A comprehensive benchmark for COVID-19 predictive modeling using electronic health records in intensive care. Patterns, 2024.
[6] Wang et al. MIMIC-Extract: A data extraction, preprocessing, and representation pipeline for MIMIC-III. ACM CHIL, 2020.
Response to Reviewer WLLW (1/2)
Dear Reviewer WLLW,
We sincerely thank the reviewer for the thoughtful review and the positive assessment! We are very happy that the reviewer finds the empirical improvements substantial across tasks and backbones :) We also appreciate the reviewer's recognition that the additional controls strengthen the case that the selection procedure contributes beyond the patient representation. We address the two remaining points below; all revisions are marked in blue in the revised paper.
W1, Q1 & Q2: Prediction Timeline and TJH Alignment
We thank the reviewer for this question, and we apologize that Appendix B.1–B.2 used the term "visit" ambiguously. We use the preprocessed cohorts of ClinicRealm [1], with MIMIC-III [2] processed by the same pipeline. As stated in Appendix B, the data are split 8:1:1 at the patient level: all admissions of the same patient are assigned to a single split, so no test patient's records enter the training casebook.
Each prediction sample is one hospital admission, which may contain several ICU stays, and a "visit" in the paper is an observation time step, i.e., one calendar day of ICU measurements within that admission (laboratory results for TJH), not a separate admission or ICU stay. "Target visit" referred both to the final observation step, whose records are withheld for mortality and LOS, and to the admission whose outcome is evaluated. The "last visit" is therefore the last observation step of the admission, not the patient's last admission. For MIMIC, ward data and records from other admissions never enter the input.
Table R1. Prediction Protocol per Task
Readmission index. The readmission index is the final ICU stay of the sampled admission, not the patient's last ICU stay in the database. The 30-day window starts at its ICU discharge and is measured from the patient's subsequent records, which MIMIC-IV links under one patient identifier with time intervals preserved [3]; these records are used only to define the label. For example, for an admission with ICU stays on days 1–3 and 5–8 and discharge on day 12, mortality and LOS use ICU days 1–3 and 5–7, with day 8 withheld as the target step. Readmission uses ICU days 1–3 and 5–8, and an ICU admission on day 20 falls within the 30-day window ending on day 38, so the label is 1.
TJH (Q2). We use the TJH dataset [4] as preprocessed in ClinicRealm following Gao et al. [5], with one hospitalization per patient, and use it only for in-hospital mortality transfer. At prediction time, the input contains all daily laboratory results except those of the final day, so TJH and MIMIC use the same calendar-day steps. TJH variables are mapped onto the MIMIC-IV ICU laboratory variables through LLM-based semantic mapping (Claude Opus 4.6), which is manually verified by two annotators with medical training and followed by unit conversion; 61 of the 73 TJH laboratory variables have a direct counterpart, and the remaining 12 are excluded. All TJH patients are used only for testing.
Interpreting the transfer results. The implementation does not use future information: no record after the prediction time, from another admission, or from the outcome enters the input. For mortality on MIMIC and TJH, we follow the retrospective structured-EHR setting of ClinicRealm [1], which uses records from the entire hospital stay. Our input is stricter: the final observation step, whose measurements are closest in time to the outcome, is additionally withheld, similar in spirit to the gap time used in MIMIC-Extract [6] to prevent temporal label leakage. All compared methods use the same records, cutoffs, and 61 mapped variables, so the transfer results compare case-selection strategies under an identical retrospective protocol in both the source and target domains. The transfer is thus evaluated under partial feature overlap, as is common across institutions, while the three key mortality predictors identified by Yan et al. [4] (lactate dehydrogenase, lymphocyte percentage, and hs-CRP) are all retained.
We have revised Appendix B.1–B.2 accordingly.
(continued below)