acceptodds
Under review as a conference paper at ICLR 2027

The Doctor’s Casebook: When LLMs Reason by Patient Analogy via Information Gain–Guided Graph Search

Abstract

Clinical reasoning over electronic health records (EHRs) remains challenging for large language models (LLMs). Inspired by clinicians who combine internal medical knowledge with experience from prior patients, we investigate Reasoning by Patient Analogy, where a frozen LLM dynamically consults a clinical casebook to ground its reasoning in prior patient experience. Our pilot study reveals three core challenges. (1) Perspective Limitation: data-side similarity does not necessarily align with the target LLM’s reasoning needs, while model-side signals are bounded by what the model itself can recognize. (2) Cohort Awareness: instance-level retrieval overlooks the population structure that organizes prior patient experience. (3) Information Aggregation: patient utility changes with the working context, causing overlapping cases to yield diminishing marginal gains as evidence accumulates. To address these challenges, we propose GraphWalker, a graph-guided framework for dynamic clinical casebook consultation. GraphWalker organizes historical patients into a clinically structured graph, uses cohort prototypes to localize relevant patient groups, and performs Context-Conditioned Greedy Search with Frontier Expansion to continually re-evaluate candidate patients by their marginal utility under the current working context, using the reduction in target-input cross-entropy as an information-gain proxy. Extensive experiments on real-world EHR benchmarks show that GraphWalker consistently outperforms strong patient-selection baselines, remains competitive with the evaluated supervised EHR predictors in-domain, and achieves stronger performance under cross-version and cross-institution transfer with only seconds-level additional inference overhead. GraphWalker further generalizes to black-box LLMs and remains compatible with downstream reasoning scaffolds. Our code is available at https://anonymous.4open.science/r/GraphWalker-4473.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, Electronic health records, Clinical prediction, Graph-based retrieval

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WLLWOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GraphWalker, a method for selecting historical patient cases as demonstrations for a frozen LLM. It organizes training patients into a graph, retrieves initial cases from relevant cohorts, and then selects demonstrations using a context-dependent information-gain proxy. Candidate scores are updated as cases are added, and graph expansion allows the search to move beyond the initial retrieval set. The evaluation covers mortality, length-of-stay, and readmission prediction, with additional experiments on transfer, different backbones, and medical question answering.

Critical strengths

The empirical improvements are substantial in several important settings. For example, with Qwen3-14B, mortality AUPRC increases from 32.23 for the strongest baseline to 59.46, while AUROC increases from 75.37 to 84.19 (Table 1). Results across different tasks and backbones give the method a reasonable empirical foundation. The appendix strengthens the case that the selection procedure contributes beyond the patient representation. Table 8 gives competing selectors access to the same SMART-based retrieval and still shows a substantial advantage for GraphWalker on mortality. Table 11 separately tests rescoring and graph expansion while fixing both the initial candidates and the final demonstration count. These are useful controls, and they provide more convincing support for the method than the main comparison alone.

Critical weaknesses

I could not fully reconstruct the clinical prediction timeline from Appendix B.1–B.2. The input consists of visits preceding the target visit, but the relationship between the “last visit” used as the target and the subsequent ICU readmission window is unclear. A valid index-stay construction may be intended, but the distinction between hospital admissions and ICU stays needs to be explicit. The observation window and variable alignment used for TJH also need more detail. These choices determine what information is available at prediction time and how the transfer results should be interpreted. Some of the smaller advantages need more cautious interpretation. For example, LLaMA readmission AUPRC is 46.37 versus 46.32 for EHR-emb, and the matched global top-9 control ties GraphWalker on readmission AUROC. Reporting means over three seeds does not show the precision of these comparisons. The large improvements remain meaningful, but the evidence for the smaller incremental benefits is less clear.

Questions for authors
  1. What exactly constitutes a visit for each task, and which stay is the readmission index? Please give a timeline showing the observed history, excluded target records, and outcome follow-up window.
  2. For TJH, which measurements are available at prediction time, and how are its temporal observations and variables aligned with the MIMIC-IV representation?
  3. Can you provide paired confidence intervals for the principal comparisons, especially against the SMART-enhanced selectors and the matched global top-9 control?
2 comments
AuthorsOpenReview ↗

Response to Reviewer WLLW (1/2)

Dear Reviewer WLLW,

We sincerely thank the reviewer for the thoughtful review and the positive assessment! We are very happy that the reviewer finds the empirical improvements substantial across tasks and backbones :) We also appreciate the reviewer's recognition that the additional controls strengthen the case that the selection procedure contributes beyond the patient representation. We address the two remaining points below; all revisions are marked in blue in the revised paper.

W1, Q1 & Q2: Prediction Timeline and TJH Alignment

We thank the reviewer for this question, and we apologize that Appendix B.1–B.2 used the term "visit" ambiguously. We use the preprocessed cohorts of ClinicRealm [1], with MIMIC-III [2] processed by the same pipeline. As stated in Appendix B, the data are split 8:1:1 at the patient level: all admissions of the same patient are assigned to a single split, so no test patient's records enter the training casebook.

Each prediction sample is one hospital admission, which may contain several ICU stays, and a "visit" in the paper is an observation time step, i.e., one calendar day of ICU measurements within that admission (laboratory results for TJH), not a separate admission or ICU stay. "Target visit" referred both to the final observation step, whose records are withheld for mortality and LOS, and to the admission whose outcome is evaluated. The "last visit" is therefore the last observation step of the admission, not the patient's last admission. For MIMIC, ward data and records from other admissions never enter the input.

Table R1. Prediction Protocol per Task

TaskObserved history (input)Prediction timeLabel and follow-up
MIMIC-III MortalityICU steps of the admission, final step withheldEnd of the last retained stepIn-hospital death
MIMIC-III LOSSame as MortalitySame as MortalityFour LOS classes (<3, 3–7, 8–14, >14 days)
MIMIC-IV ReadmissionAll ICU steps up to discharge from the index ICU stayIndex ICU discharge30-day readmission outcome after index ICU discharge
TJH MortalityDaily laboratory steps of the hospitalization, final day withheldEnd of the last retained stepIn-hospital death

Readmission index. The readmission index is the final ICU stay of the sampled admission, not the patient's last ICU stay in the database. The 30-day window starts at its ICU discharge and is measured from the patient's subsequent records, which MIMIC-IV links under one patient identifier with time intervals preserved [3]; these records are used only to define the label. For example, for an admission with ICU stays on days 1–3 and 5–8 and discharge on day 12, mortality and LOS use ICU days 1–3 and 5–7, with day 8 withheld as the target step. Readmission uses ICU days 1–3 and 5–8, and an ICU admission on day 20 falls within the 30-day window ending on day 38, so the label is 1.

TJH (Q2). We use the TJH dataset [4] as preprocessed in ClinicRealm following Gao et al. [5], with one hospitalization per patient, and use it only for in-hospital mortality transfer. At prediction time, the input contains all daily laboratory results except those of the final day, so TJH and MIMIC use the same calendar-day steps. TJH variables are mapped onto the MIMIC-IV ICU laboratory variables through LLM-based semantic mapping (Claude Opus 4.6), which is manually verified by two annotators with medical training and followed by unit conversion; 61 of the 73 TJH laboratory variables have a direct counterpart, and the remaining 12 are excluded. All TJH patients are used only for testing.

Interpreting the transfer results. The implementation does not use future information: no record after the prediction time, from another admission, or from the outcome enters the input. For mortality on MIMIC and TJH, we follow the retrospective structured-EHR setting of ClinicRealm [1], which uses records from the entire hospital stay. Our input is stricter: the final observation step, whose measurements are closest in time to the outcome, is additionally withheld, similar in spirit to the gap time used in MIMIC-Extract [6] to prevent temporal label leakage. All compared methods use the same records, cutoffs, and 61 mapped variables, so the transfer results compare case-selection strategies under an identical retrospective protocol in both the source and target domains. The transfer is thus evaluated under partial feature overlap, as is common across institutions, while the three key mortality predictors identified by Yan et al. [4] (lactate dehydrogenase, lymphocyte percentage, and hs-CRP) are all retained.

We have revised Appendix B.1–B.2 accordingly.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer WLLW (2/2)

W2 & Q3: Paired Confidence Intervals for the Principal Comparisons

We thank the reviewer for this suggestion and agree that means over three runs do not show the precision of the comparisons. We therefore report paired bootstrap confidence intervals for the comparisons highlighted by the reviewer. Consistent with the reviewer's assessment, the intervals support the larger gains, including the advantage over the strongest SMART-enhanced selector on all three metrics and the mortality AUPRC gain from cohort initialization, whereas the smaller differences remain uncertain.

Experimental setup. For each comparison, we resample test patients with replacement. In each resample, both methods are evaluated on the same patients; each metric is computed for each of the three runs and averaged over runs, consistent with the reported means, and we take the difference between GraphWalker and the comparator. We report the difference on the full test set together with the 2.5th and 97.5th percentiles of the resampled differences. F1 uses the same fixed decision threshold as in the main results.

Table R2. Paired Bootstrap Differences (GraphWalker − Comparator)

Differences in percentage points, with pointwise 95% confidence intervals in brackets; positive values favor GraphWalker. Bold: the interval lies entirely above zero.

SettingComparatorAUROCAUPRCF1
MIMIC-III Mortality, LLaMA-3.1-8B (Table 8)Delta-KNN + SMART+9.40 [5.42, 13.67]+6.97 [1.31, 13.24]+6.07 [1.58, 10.20]
MIMIC-III Mortality, Qwen3-14B (Table 10)w/o Cohort (global top-9)†+2.03 [−0.67, 4.62]+8.44 [3.65, 13.12]+1.69 [−2.63, 6.00]
MIMIC-IV Readmission, Qwen3-14B (Table 10)w/o Cohort (global top-9)†0.00 [−1.65, 1.78]+0.84 [−1.63, 3.40]+1.00 [−1.33, 3.44]
MIMIC-IV Readmission, LLaMA-3.1-8B (Table 1)EHR-emb+4.57 [0.89, 8.30]+0.05 [−4.81, 4.86]+1.73 [−1.61, 5.11]

† The Table 10 comparator uses global top-9 initialization followed by the same graph expansion and early-stopping rules as GraphWalker.

Analysis & Conclusion.

First, the advantage over the strongest SMART-enhanced selector is supported across all three metrics. On MIMIC-III Mortality with LLaMA-3.1-8B, GraphWalker improves AUROC / AUPRC / F1 over Delta-KNN + SMART by 9.40 / 6.97 / 6.07 points, and all three confidence intervals lie entirely above zero. The remaining SMART-enhanced selectors trail GraphWalker by larger margins on all three metrics (Table 8). Since the comparator also uses SMART-based retrieval, this provides further support for the contribution of GraphWalker's selection procedure beyond access to an EHR-specific encoder.

Second, cohort initialization provides a clear mortality AUPRC gain with the initial candidate count matched. Against global top-9 initialization with Qwen3-14B, GraphWalker improves AUPRC by 8.44 points [3.65, 13.12]. Both methods start from nine candidates and retain the same subsequent search procedure, supporting the value of cohort-based entry points for mortality prediction. The AUROC and F1 intervals include zero, so the evidence for this incremental benefit is strongest on AUPRC.

Third, readmission results distinguish a supported AUROC gain from uncertain smaller differences. With LLaMA-3.1-8B, GraphWalker improves AUROC over EHR-emb by 4.57 points [0.89, 8.30]. The near-tied AUPRC difference of 0.05 points [−4.81, 4.86] and the F1 interval do not establish additional gains. Likewise, all three intervals for Qwen3-14B against global top-9 initialization include zero. We therefore interpret these smaller margins with caution, while the larger gains above remain supported.

We thank the reviewer again for the constructive suggestions, which helped us make the prediction protocol explicit and the interpretation of our results more precise. We hope these responses address the reviewer's concerns.

References

[1] Zhu et al. ClinicRealm: Re-evaluating large language models with conventional machine learning for non-generative clinical prediction tasks. npj Digital Medicine, 2026.

[2] Johnson et al. MIMIC-III, a freely accessible critical care database. Scientific Data, 2016.

[3] Johnson et al. MIMIC-IV, a freely accessible electronic health record dataset. Scientific Data, 2023.

[4] Yan et al. An interpretable mortality prediction model for COVID-19 patients. Nature Machine Intelligence, 2020.

[5] Gao et al. A comprehensive benchmark for COVID-19 predictive modeling using electronic health records in intensive care. Patterns, 2024.

[6] Wang et al. MIMIC-Extract: A data extraction, preprocessing, and representation pipeline for MIMIC-III. ACM CHIL, 2020.

Reviewer_T52oOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how LLMs should select historical cases for EHR clinical reasoning and proposes GraphWalker: it first uses a pretrained EHR encoder to compress patients into vectors, builds a kNN patient graph, and uses Leiden to partition cohorts; it matches the target patient with cohort prototypes and selects anchor patients within relevant cohorts; then, on the frontier formed by the anchors, it iteratively selects cases according to the marginal decrease in cross-entropy on the target input, expands along graph neighbors, and applies early stopping; finally, the selected cases are passed to a frozen LLM as few-shot examples. Its main contributions are threefold: first, it decomposes clinical case review into three problems: perspective limitation, cohort awareness, and information aggregation; second, it proposes a case selection framework of “cohort guidance + context-conditioned marginal information gain,” combining data-side population structure with model-side utility; third, extensive experiments demonstrate that the framework is effective under multi-task, multi-backbone, cross-domain, black-box, and reasoning scaffold settings. For validation, the authors evaluate mortality, readmission, and LOS on MIMIC-III, MIMIC-IV, and TJH, comparing against zero-shot, random, and various data-side/model-side selection baselines. The main results report average gains in AUROC/AUPRC/F1 and other metrics; the method is competitive with supervised predictors in-domain and shows stronger transfer across versions and institutions. Ablations show that EHR representation, cohorts, context-conditioned rescoring, frontier expansion, and early stopping all contribute; a global top-9 control rules out the explanation of “more candidates alone”; black-box self-evaluation, prompt sensitivity, calibration, and MedQA/CMB/CMB-Clin provide supplementary evidence. Overall, this is a paper with a reasonable method combination and complete experiments but moderate novelty.

Critical strengths

This paper proposes a hierarchical mechanism for retrieving similar cases, combined with context-conditioned greedy search filtering, which can effectively reduce the selection of duplicate cases and irrelevant case references.

Critical weaknesses

When computing cohort prototypes, the mean-pooling method is used, which underestimates the internal heterogeneity of the cohort and minority but important clinical patterns. When the cohort is multimodal or long-tailed, mean prototypes may not be representative, thereby affecting cohort retrieval.

Questions for authors
  1. When computing cohort prototypes, the mean-pooling method is used, which underestimates the internal heterogeneity of the cohort and minority but important clinical patterns. When the cohort is multimodal or long-tailed, mean prototypes may not be representative, thereby affecting cohort retrieval.

Q1.1:What is the distribution of similarity between cohort members and the prototype? Please report, for each cohort, the median, interquartile range, and lower-quantile tail of the member–prototype cosine similarity, as well as whether multimodality or a long tail is present.

Q1.2: In the pipeline, does the prototype serve only as a coarse filter, or does it also participate in subsequent scoring? Please confirm whether zm\mathbf{z}_m is used only for top-KcK_c cohort retrieval and does not enter anchor ranking or the final ΔH\Delta H scoring; also clarify whether L2 normalization is performed before averaging.

Q1.3: Compare mean-poolinfg with medoid, multi-prototype, and attention pooling for computing prototypes.

3 comments
AuthorsOpenReview ↗

Response to Reviewer T52o (1/3)

Dear Reviewer T52o,

We sincerely thank the reviewer for the detailed summary and the positive assessment! We are delighted that the reviewer recognizes that the hierarchical case-retrieval mechanism, combined with context-conditioned greedy search, effectively reduces the selection of duplicate and irrelevant cases. We also appreciate the reviewer's recognition that the framework reasonably combines data-side population structure with model-side utility, and that the experiments are complete, covering multiple tasks, backbones, and cross-domain settings. We address the reviewer's questions on cohort prototypes below; all revisions are marked in blue in the revised paper.

W1 & Q1.2: Role of the Cohort Prototype in the Pipeline

We thank the reviewer for this question. GraphWalker follows a coarse-to-fine design: the prototypes first locate a few clinically relevant cohorts of patients similar to the target, and the ΔH\Delta H-guided search then walks on the patient graph from anchors in these cohorts to select the most informative demonstrations. Since the prototype is meant to locate relevant cohorts rather than to rank individual patients, it serves only as a coarse filter. Specifically:

  • Cohort retrieval is the only step that uses zm\mathbf{z}_m. As defined in Section 3.2, the top-KcK_c cohorts are retrieved by the cosine similarity between the target patient and each prototype. Comparing the target with a small set of compact prototypes efficiently narrows the search to relevant subpopulations, and retrieving several cohorts provides multiple entry points, since a patient may share clinical patterns with more than one subpopulation.
  • Anchor ranking does not use the prototype. Within each retrieved cohort, anchors are ranked by the similarity between the target patient and each individual member, sim(h(q),hi)\mathrm{sim}(\mathbf{h}^{(q)},\mathbf{h}_i). Anchors therefore reflect the members closest to the target rather than to the cohort center, so members with less typical patterns can still be selected when they resemble the target.
  • ΔH\Delta H scoring does not use the prototype. The scores are computed from the LLM's cross-entropy on the target record, so the final selection reflects how informative each case is for the target given the demonstrations already selected. Graph-neighbor expansion can also reach patients outside the retrieved cohorts, allowing the search to move across cohort boundaries.
  • Regarding normalization, the prototype averages the original member embeddings without prior L2 normalization. Prompted by this question, we also include an L2-normalized variant in Q1.3, which retrieves nearly the same cohorts (Jaccard 0.98) with comparable performance.

The prototype therefore determines only the entry cohorts, whose sensitivity we examine in Q1.1 and Q1.3 below.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer T52o (2/3)

Q1.1: Member–Prototype Similarity Distribution

We examine all 19 cohorts, covering 8,160 training cases, in the representative MIMIC-III Mortality setting. As the reviewer anticipated, cohorts differ in how closely their members align with the mean prototype, and several cohorts show a longer lower-similarity tail. The dip test does not detect significant multimodality in the one-dimensional similarity distributions, although such a test cannot rule out heterogeneity in the embedding space; we therefore further compare alternative prototypes in Q1.3.

Experimental setup. We use the frozen SMART checkpoint and Leiden partition from one main-experiment run. For each cohort, we compute the cosine similarity between every member embedding and its default mean prototype (Q1.2). We report the cohort size, median, interquartile range (IQR = Q75 − Q25), and fifth and ninety-fifth percentiles (P5 and P95) for each cohort in Table R1.

Distribution diagnostics. We characterize the lower-similarity tail using P5 and its distance from the median, with P95 providing the corresponding upper-tail reference. To assess multimodality, we apply Hartigan's dip test to each cohort's member–prototype similarities, estimate p-values by Monte Carlo simulation, and apply Benjamini–Hochberg correction across the 19 cohorts, using an adjusted-p threshold of 0.05. Table R1 reports the raw p-values; the corrected results are given below it.

Table R1. Member–Prototype Similarity by Cohort on MIMIC-III Mortality

Training casebook. Cohorts are indexed by size; N denotes the number of cases in each cohort. Similarity statistics are rounded to three decimal places and p-values to two. Raw p denotes the Monte Carlo p-value for Hartigan's dip test.

CohortNMedianIQRP5P95Raw p
C18210.6610.1360.4000.7780.39
C27550.6370.1410.3770.7641.00
C37070.7540.0490.6910.8081.00
C45970.7690.0970.6450.8590.54
C55710.6810.0940.4570.7741.00
C65410.7660.0620.6760.8370.99
C74820.6960.0690.4040.7570.97
C84680.7640.0910.6360.8560.98
C94500.7660.0790.6700.8631.00
C104040.6890.0570.6190.7521.00
C113900.6300.1690.3880.7870.96
C123100.6920.1140.4530.7890.99
C133070.7300.1160.5840.8300.88
C143030.7350.0700.6480.8070.93
C152610.6930.1120.5450.8060.96
C162290.7110.1110.5850.8270.83
C172200.7430.0580.6790.7970.51
C181720.7470.0900.6350.8410.91
C191720.6560.1680.3950.7710.91

All 19 Benjamini–Hochberg-adjusted p-values equal 1.00; no cohort meets the 0.05 significance threshold.

Analysis & Conclusion.

First, member–prototype alignment varies across cohorts. Cohort medians range from 0.630 to 0.769 and IQRs from 0.049 to 0.169; for example, C3 is concentrated (median 0.754, IQR 0.049), whereas C11 is more dispersed (median 0.630, IQR 0.169).

Second, several cohorts have a longer lower-similarity tail. In C7, the median is 0.696 and P95 is 0.757, but P5 falls to 0.404; C1, C2, C5, C11, C12, and C19 show a similar lower-side asymmetry, with median − P5 exceeding 0.2. We agree with the reviewer that the mean prototype represents these lower-similarity members less closely. In GraphWalker, however, the prototype only selects the entry cohorts (Q1.2), so members in these tails can still be selected as anchors or reached by graph expansion when they resemble the target.

Third, the dip tests do not detect significant multimodality in the similarity distributions. Raw p-values range from 0.39 to 1.00, and none of the 19 cohorts rejects unimodality after Benjamini–Hochberg correction. Since this test concerns only the one-dimensional similarity distribution, it cannot rule out heterogeneity in the embedding space, which motivates the prototype comparison in Q1.3.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer T52o (3/3)

Q1.3: Alternative Prototype Constructions

Richer prototype constructions can further help: the multi-prototype and attention-pooling variants suggested by the reviewer yield small improvements on all six metrics. At the same time, GraphWalker shows limited sensitivity to prototype construction, with all alternatives staying within 0.72 points of the default mean prototype.

Experimental setup. Following the reviewer's suggestion, we compare the default mean prototype with the medoid, multi-prototype, and attention pooling, as well as the L2-normalized mean discussed in Q1.2, on MIMIC-III Mortality and MIMIC-IV Readmission, using Qwen3-14B as in Tables 10 and 11. Within each run, all variants share the same frozen SMART checkpoint, Leiden partition, training casebook, test split, task prompts, and decoding settings. They also use the default configuration (kg=8k_g=8, Kc=3K_c=3, Ka=3K_a=3, at most three demonstrations, early stopping enabled). Only the prototype used for top-KcK_c cohort retrieval changes; anchor selection within each retrieved cohort, context-conditioned ΔH search, and graph-neighbor expansion are unchanged.

  • Mean (default) averages the original member embeddings.
  • L2-normalized mean L2-normalizes each member embedding before averaging.
  • Medoid uses the member with the highest average cosine similarity to the other members of its cohort.
  • Multi-prototype partitions each cohort into m=3m=3 sub-clusters with kk-means and scores the cohort by the maximum cosine similarity between the target patient and the sub-cluster means; cohorts with fewer than mm members use a single mean.
  • Attention pooling forms a target-specific prototype zm(q)=∑i∈Cmαihi\mathbf{z}_m(q)=\sum_{i\in\mathcal{C}_m}\alpha_i\mathbf{h}_i, with αi=softmaxi(cos⁡(h(q),hi)/τ)\alpha_i=\mathrm{softmax}_i\big(\cos(\mathbf{h}^{(q)},\mathbf{h}_i)/\tau\big) and τ=0.1\tau=0.1. This similarity-based weighting introduces no additional learned parameters.

The values of mm and τ\tau are fixed in advance and not tuned. To show how much each variant changes cohort retrieval, we also report the average Jaccard overlap between its top-KcK_c cohorts and those retrieved with the mean prototype, computed over test patients.

Table R2. Performance with Alternative Prototype Constructions

Qwen3-14B; means over three runs. AUROC / AUPRC / F1 (%). Jaccard: average overlap between the retrieved top-KcK_c cohorts and those of the default mean prototype.

PrototypeMIMIC-III MortalityJaccardMIMIC-IV ReadmissionJaccard
Mean (default)†84.19 / 59.46 / 58.241.0075.91 / 60.61 / 58.011.00
L2-normalized mean84.27 / 59.63 / 58.180.9875.83 / 60.54 / 58.090.98
Medoid83.74 / 58.88 / 57.820.5975.66 / 60.08 / 57.730.55
Multi-prototype (m=3m=3)84.48 / 60.18 / 58.650.9076.22 / 61.05 / 58.440.91
Attention pooling (τ=0.1\tau=0.1)84.36 / 59.97 / 58.410.8276.08 / 61.19 / 58.320.86

† Same runs as GraphWalker with Qwen3-14B in Table 1.

Analysis & Conclusion.

First, richer prototypes yield small improvements. Multi-prototype and target-specific attention pooling, both designed to reflect within-cohort heterogeneity, improve all six metrics by 0.17–0.72 points, in line with the reviewer's view that richer cohort representations can help.

Second, overall performance remains stable across prototype constructions. On both tasks, every alternative changes AUROC, AUPRC, and F1 by at most 0.72 points relative to the mean prototype. The L2-normalized mean retrieves almost the same cohorts (Jaccard 0.98) and changes each metric by at most 0.17 points.

Third, different entry cohorts can yield similar prediction performance. The medoid changes the retrieved cohorts the most (Jaccard 0.59 and 0.55), yet lowers each metric by at most 0.58 points. This is consistent with the coarse-filter role of the prototype (Q1.2): anchors and subsequent candidates are selected at the patient level.

We retain the mean prototype as the default for simplicity and include these alternatives in Appendix H.2 of the revised paper as optional refinements.

We thank the reviewer again for the insightful questions on cohort heterogeneity, which helped us clarify the role of cohort prototypes and evaluate richer alternatives. We hope these responses address the reviewer's concerns.

Reviewer_LBByOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies selection of historical patient examples for LLM prediction from longitudinal EHRs. GraphWalker builds a patient graph using an EHR encoder, retrieves cohort anchors, and greedily selects examples by their marginal reduction in target-input token cross-entropy. It expands the search through neighbors of selected cases and stops at a demonstration budget or non-positive estimated gain. Experiments report improvements on MIMIC outcome-prediction tasks and MIMIC-IV-to-TJH transfer, supported by encoder, cohort, and search-component controls.

Critical strengths
  1. The improvement survives substantive controls. SMART-enhanced baselines, multiple EHR encoders, matched initial candidate count, and a factorial search ablation support an algorithmic benefit beyond generic text retrieval or encoder access. This is the principal reason to accept the methodological contribution.
  2. The evaluation addresses practical predictive value. It includes supervised comparisons, a held-out institution, calibration, and an explicit token/call/latency profile. These make the empirical contribution more useful than an accuracy-only demonstration, within the retrospective setting.
Critical weaknesses
  1. The prediction timeline is insufficiently specified. The target is called the last visit, yet readmission is labeled after the target ICU stay; the relationship between this target, historical inputs, follow-up, and TJH is unclear. This matters for rejection if the implementation uses future information or evaluates a different endpoint than claimed.

  2. The graph-specific benefit is not isolated at equal scoring cost. The factorial ablation fixes initial candidates and final demonstrations but explicitly allows different scoring workloads. A matched-budget flat SMART candidate search using the same conditional scorer is needed to determine whether graph exploration earns its complexity. The existing results establish the combination’s utility more clearly than its efficiency advantage.

Questions for authors

Please give an example timeline and the cohort-extraction rules for each dataset/task: what counts as a visit, when the prediction is made, which records enter the encoder and prompt, how the target is selected, and how 30-day readmission follow-up is observed despite calling it the last visit. How does this construction apply to TJH?

3 comments
AuthorsOpenReview ↗

Response to Reviewer LBBy (1/3)

Dear Reviewer LBBy,

We sincerely thank the reviewer for the careful reading and the positive assessment! It is very encouraging to us that the reviewer finds the improvement to survive substantive controls (SMART-enhanced baselines, multiple EHR encoders, matched initial candidate count, and the factorial search ablation), supporting an algorithmic benefit beyond generic text retrieval or encoder access. We also appreciate the reviewer's recognition that the evaluation addresses practical predictive value through supervised comparisons, a held-out institution, calibration, and an explicit token/call/latency profile. We address the two remaining points below; all revisions are marked in blue in the revised paper.

W1 & Q1: Prediction Timeline and Cohort Construction

We thank the reviewer for raising this point, and we apologize that Appendix B used the term "visit" ambiguously. We use the preprocessed cohorts of ClinicRealm [1], with MIMIC-III [2] processed by the same pipeline. As stated in Appendix B, the data are split 8:1:1 at the patient level: all admissions of the same patient are assigned to a single split, so no test patient's records enter the training casebook.

Each prediction sample is one hospital admission, and a "visit" in the paper is an observation time step, i.e., one calendar day of ICU measurements within that admission (laboratory results for TJH), not a separate admission or ICU stay. For MIMIC, ward data and records from other admissions never enter the input, and the EHR encoder and the prompt use the same records. "Target visit" referred both to the final observation step, whose records are withheld for mortality and LOS, and to the admission whose outcome is evaluated. The "last visit" is therefore the last observation step of the admission, not the patient's last admission, so the 30-day readmission follow-up is observable: MIMIC-IV links all ICU stays of a patient under one identifier, and its date shifting preserves time intervals within each patient [3], so the window after index ICU discharge is measured directly from the patient's later records.

Table R1. Prediction Protocol per Task

TaskInput (encoder and prompt)Prediction timeLabel
MIMIC-III MortalityICU steps of the admission, final step withheldEnd of the last retained stepIn-hospital death
MIMIC-III LOSSame as MortalitySame as MortalityFour LOS classes (<3, 3–7, 8–14, >14 days)
MIMIC-IV ReadmissionAll ICU steps up to discharge from the index ICU stay (the final ICU stay of the admission)Index ICU discharge30-day readmission outcome after index ICU discharge
TJH Mortality (transfer)Daily laboratory steps of the hospitalization, final day withheldEnd of the last retained stepIn-hospital death

Example. For an admission with ICU stays on days 1–3 and 5–8 and discharge on day 12, mortality and LOS use ICU days 1–3 and 5–7. Readmission uses ICU days 1–3 and 5–8, and an ICU admission on day 20 falls within the follow-up window ending on day 38, so the label is 1.

On future information. The implementation does not use future information: no record after the prediction time, from another admission, or from the outcome enters the encoder or the prompt. The ΔH scorer scores only the target-record tokens, without the target label, and readmission is predicted at index ICU discharge. For mortality on MIMIC and TJH, we follow the retrospective structured-EHR setting of ClinicRealm [1], which uses records from the entire hospital stay. Our input is stricter: the final observation step, whose measurements are closest in time to the outcome, is additionally withheld, similar in spirit to the gap time used in MIMIC-Extract [4] to prevent temporal label leakage. All compared methods, including the supervised baselines, use the same records and cutoffs, ensuring consistent information access.

TJH. We use the TJH dataset [5] as preprocessed in ClinicRealm following Gao et al. [6], with one hospitalization per patient and daily laboratory measurements. TJH is used only for in-hospital mortality transfer, and all TJH patients serve only as test cases. Since TJH has no post-discharge follow-up, no readmission index or follow-up window is defined for it.

We have revised Appendix B.1–B.2 accordingly.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer LBBy (2/3)

W2: Graph Exploration under Matched Candidate-Evaluation Budgets

We thank the reviewer for this constructive suggestion. We agree that Table 11 controls the initial candidates and final demonstration count, but allows different candidate-scoring workloads. We therefore compare GraphWalker with two flat SMART searches under matched per-patient, per-round candidate-evaluation budgets. GraphWalker achieves higher mean performance than both controls on all three metrics in both tasks, supporting the value of graph-guided exploration beyond additional candidate evaluations. We have added this comparison to Appendix H.1 of the revised paper.

Experimental setup. We use Qwen3-14B on MIMIC-III Mortality and MIMIC-IV Readmission, following Table 11. Within each run, all variants share the same frozen SMART checkpoint, training casebook, test split, EHR serialization, task prompts, and decoding settings. All select exactly three demonstrations, with early stopping disabled, and use the same context-conditioned ΔH scorer and selection-order formatting:

  • GraphWalker (w/o Early Stop) starts from nine anchors (three patients from each of three retrieved cohorts) and expands the candidate pool through each selected patient's graph neighbors, with kg=8k_g=8.
  • Flat SMART (shared anchors) starts from exactly the same nine anchors, but replaces graph-neighbor expansion with additional candidates retrieved from the full training casebook by SMART cosine similarity to the target patient.
  • Flat SMART (global top-9) starts from the nine globally most similar training patients under SMART cosine similarity and uses the same flat replenishment procedure. It uses neither cohort retrieval nor graph-neighbor expansion.

Budget matching. For each test patient in each run, we first record the number of candidates evaluated in each of GraphWalker's three rounds. Both flat variants match these counts round by round: after each selection, they retain the unselected candidates and fill the next round's remaining slots from the fixed, target-specific SMART ranking, excluding selected cases and cases already in the pool. Every round re-scores all current candidates conditioned on that variant's own selected demonstrations and adds the highest-ΔH candidate. One evaluation means scoring one candidate under one working context; evaluating it again in a later round counts again. The flat variants receive only GraphWalker's per-round counts, without access to its subsequent candidate identities, scores, or selections.

Table R2. Performance under Matched Candidate-Evaluation Budgets

AUROC / AUPRC / F1 (%); means over the same three runs. All variants select exactly three demonstrations. Bold / italic: best / second-best mean.

MethodMIMIC-III MortalityMIMIC-IV Readmission
GraphWalker (w/o Early Stop)†81.34 / 48.66 / 55.9975.53 / 59.13 / 57.38
Flat SMART (shared anchors)78.16 / 47.02 / 54.9872.08 / 54.79 / 54.86
Flat SMART (global top-9)77.24 / 46.38 / 54.2172.61 / 55.26 / 55.31

† GraphWalker (w/o Early Stop) results are reproduced from Table 11.

Analysis & Conclusion.

First, graph expansion contributes beyond the number of candidates evaluated. With identical initial anchors, GraphWalker exceeds Flat SMART (shared anchors) by 3.18 / 1.64 / 1.01 points (AUROC / AUPRC / F1) on mortality and 3.45 / 4.34 / 2.52 points on readmission. Both methods retain context-conditioned re-scoring and evaluate the same number of candidates in every round. This comparison supports the benefit of using selected patients' graph neighbors to guide subsequent candidate exploration.

Second, the complete search outperforms a fully flat SMART alternative, regardless of initialization. Against Flat SMART (global top-9), GraphWalker improves AUROC / AUPRC by 4.10 / 2.28 points on mortality and 2.92 / 3.87 points on readmission, with higher F1 on both tasks. This flat control retrieves from the entire training casebook and uses the same conditional scorer and evaluation budget. Since GraphWalker exceeds both flat variants, whether they start from the same cohort anchors or from the global nearest neighbors, the gains are not explained by initialization alone.

Third, the improvement comes at little additional cost. Since the number of LLM candidate evaluations is matched, the additional cost of GraphWalker mainly lies in the one-time offline graph construction (13.6 min on MIMIC-III Mortality, Table 17) and neighbor lookup at inference. Under the same evaluation budget, graph exploration thus yields higher performance, suggesting that it earns its added complexity in these settings.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer LBBy (3/3)

We thank the reviewer again for the constructive suggestions, which helped us clarify the prediction protocol and strengthen the analysis of the search design. We hope these responses address the reviewer's concerns.

References

[1] Zhu et al. ClinicRealm: Re-evaluating large language models with conventional machine learning for non-generative clinical prediction tasks. npj Digital Medicine, 2026.

[2] Johnson et al. MIMIC-III, a freely accessible critical care database. Scientific Data, 2016.

[3] Johnson et al. MIMIC-IV, a freely accessible electronic health record dataset. Scientific Data, 2023.

[4] Wang et al. MIMIC-Extract: A data extraction, preprocessing, and representation pipeline for MIMIC-III. ACM CHIL, 2020.

[5] Yan et al. An interpretable mortality prediction model for COVID-19 patients. Nature Machine Intelligence, 2020.

[6] Gao et al. A comprehensive benchmark for COVID-19 predictive modeling using electronic health records in intensive care. Patterns, 2024.

Comments on the paper

4 comments
AuthorsOpenReview ↗

A Humble Response to All Reviewers

We would like to express our sincere gratitude to all reviewers for their thoughtful, constructive, and insightful comments! We are encouraged that the reviewers find that the improvements of GraphWalker survive substantive controls (Reviewer LBBy) and are substantial across tasks and backbones (Reviewer WLLW), that the evaluation addresses practical predictive value (Reviewer LBBy), and that the hierarchical case retrieval with context-conditioned search effectively reduces duplicate and irrelevant cases, supported by complete experiments (Reviewer T52o). Your feedback has been invaluable in improving the clarity and rigor of our work.

In response to your suggestions, we have conducted additional experiments and carefully revised the manuscript. The new analyses are placed in Appendix H, and all revisions are highlighted in blue:

  • Prediction timeline and cohort construction (Reviewers LBBy and WLLW; Appendix B.1–B.2 and Appendix F). We clarified the definitions of prediction samples, visits, and target records, the readmission index stay and follow-up window, and the observation window and variable alignment for TJH.
  • Graph exploration under matched candidate-evaluation budgets (Reviewer LBBy; Appendix H.1). We compared GraphWalker with two flat SMART searches that use the same conditional scorer and evaluate the same number of candidates per patient and round.
  • Alternative cohort prototypes (Reviewer T52o; Appendix H.2). We compared the default mean prototype with the medoid, multi-prototype, and attention-pooling variants, as well as an L2-normalized mean.

In addition, the individual responses report the member–prototype similarity distributions for all cohorts (Reviewer T52o) and paired bootstrap confidence intervals for the principal comparisons (Reviewer WLLW).

We sincerely hope these revisions adequately address your concerns, and we appreciate the time and care you have devoted to reviewing our work. We look forward to your feedback.

Best regards,

All Authors of Submission 20214

Thank you for your responses, I will maintain my original score.

Dear Reviewer WLLW,

Thank you so much for reading our responses and for your support of our work! ! Your questions on the timeline and the confidence intervals really helped us make the paper clearer, and we appreciate the time you spent on it. Thanks again :)

Best regards,

Authors of Submission 20214

AuthorsOpenReview ↗

General Rebuttal Summary

We sincerely thank all reviewers for their general positive assessments of our work and their insightful, constructive comments.

We are grateful to all three reviewers, whose assessments highlight both GraphWalker’s grounding in how clinicians reason from prior patient cases and its empirical gains across clinical prediction tasks and LLM backbones. Reviewer LBBy saw the controlled comparisons as evidence that GraphWalker’s gains reflect how it selects cases, not just the encoder it uses. Reviewer T52o highlighted how hierarchical retrieval and context-conditioned search can reduce redundant or irrelevant cases. Reviewer WLLW noted substantial gains across several settings and found the controlled comparisons more convincing than the main results alone. They also highlight our paper’s clarity, experimental breadth, and informative ablations.

In response, we conducted additional experiments, added new analyses, and provided clarifications to strengthen the manuscript further. Major revisions include:

1. What graph exploration adds at a matched scoring budget (Reviewer LBBy; Appendix H.1, Table 19)

With the same starting cases, scorer, and candidate-evaluation budget, GraphWalker outperformed flat retrieval on both mortality and readmission. GraphWalker’s patient graph links case selection to retrieval: each selected case both informs the prediction and brings its neighbors into consideration for the next round.

2. Prediction timeline and transfer to a held-out hospital (Reviewers LBBy and WLLW; Appendices B.1–B.2 and F)

We clarified the definitions of prediction samples, visits, and target records, the readmission index stay and follow-up window, and the observation window and variable alignment for TJH. We thank Reviewers LBBy and WLLW for helping us make the evaluation protocol explicit. Reviewer WLLW maintained their positive rating.

3. How much the cohort prototype matters (Reviewer T52o; Appendix H.2, Table 20)

We compared the default mean prototype with the reviewer’s suggested alternatives for choosing where graph search begins. Multi-prototype and attention pooling yielded modest gains, while the simple mean prototype remained competitive on both tasks.

4. Which performance gains the paired intervals support (Reviewer WLLW; reported in our response)

We reported paired bootstrap intervals for the principal comparisons. On MIMIC-III mortality prediction, the intervals favor GraphWalker over the strongest SMART-based selector on all three metrics, even though both use the same encoder.

Finally, we briefly summarize the discussion stage.

All three ratings remain positive (3 from each reviewer). Reviewer WLLW explicitly confirmed their rating during the discussion.

We thank the reviewers, ACs, SACs and PCs again for their time and valuable feedback, and we remain available for any further questions or discussion.

Warmest regards,

The Authors of Submission 20214

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.