acceptodds
Under review as a conference paper at ICLR 2027

: Prototype-Guided Retrieval of Longitudinal Electronic Health Records for Clinical Prediction Models

80 views

Abstract

Electronic Health Records (EHR) contain rich longitudinal patient information and are widely used in predictive modeling applications. However, effectively leveraging historical data remains challenging due to long trajectories, heterogeneous events, temporal irregularity, and the varying relevance of past clinical context. Existing approaches often rely on fixed windows or uniform aggregation, which can obscure clinically important signals. In this work, we introduce , a retrieval-based framework that dynamically integrates the most relevant patient history consisting of diverse clinical event types. We propose a prototype-guided retrieval module that acts as an alignment mechanism and estimates the relevance of retrieved historical chunks with respect to a given prediction task, guiding the model towards the most informative context. Across multiple clinical prediction tasks and two benchmark datasets, consistently outperforms state-of-the-art EHR-based and transformer-based baselines. Furthermore, is model-agnostic, as integrating it with various backbone models yields substantial performance gains. Overall, establishes a novel direction for modeling long-range clinical context to improve downstream performance via retrieval.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
EHR, Retrieval, RAG, Prediction Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xmevOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper recasts long-horizon EHR modeling as retrieval. A patient's history is chunked along four granularities (event, time, visit, care stage), embedded with a RoBERTa encoder, stored in a FAISS index, and top-M chunks are retrieved by cosine similarity. A prototype-guided alignment operator soft-reweights chunks by agreement between query and chunk prototype assignments, and a two-layer transformer fuses the result. Evaluation covers four MIMIC-IV ICU tasks and nine EHRSHOT tasks with a CLMBR backbone, claiming improvement in every backbone/task/metric cell tested.

Critical strengths

Leakage hygiene is better than typical here: pretraining uses only the training split, splits are patient-disjoint, normalization is fit on training data only, and the paper explicitly declines significance claims against EHRSHOT long-context baselines it cannot match. Confidence intervals are reported for all MIMIC numbers.

Critical weaknesses
  • The headline numbers cannot be reproduced from the paper's own tuning and ablation tables: Appendix S4 reports Ts = 0.22 for the one-year task, outside the search space {0.1, 0.15, 0.2, 0.25} declared in S3, and S13 shows the best one-year setting at Ts = 0.20 (0.853 AUROC / 0.564 AUPRC).
  • The in-hospital task is inconsistent the same way: S4 uses 0.20 while S13 reports 0.25 as best (0.950 / 0.736).
  • Table 5 reports ICU prototype entropies of 6.198 and 6.232, which exceed the maximum possible for the 64-prototype ICU configuration (ln 64 = 4.159); values near ln 512 = 6.238 suggest a different prototype count was used, and since the ablation never reproduces the headline values I could not determine which configuration produced them.
  • No seeds or error bars are reported anywhere, and the prototype-attribution gain (AUROC 0.008-0.015) is smaller than the spread of the prototype-count sweep itself (0.018 in S12), so the benefit of the central mechanism is within run-to-run variation.
  • On EHRSHOT the paper reports no confidence intervals and no significance tests, and uses quoted rather than re-run baselines for eight of nine headline tasks; Section S10 reports p-values from 0.0098 to 0.0355 across roughly 112 tests with no multiplicity correction.
  • Table 5 shows prototype assignments operating at about 99% of maximum entropy, meaning the weights are nearly uniform, which sits awkwardly with the claim that prototypes meaningfully guide retrieval.
Questions for authors
  1. Which exact configuration produced the headline 0.750/0.141 and 0.853/0.564 results? Please reconcile S4, S12 and S13, including the Ts = 0.22 value.
  2. What prototype count and model size produced the Table 5 ICU entropies, and can Table 5 be reproduced from a stated configuration?
  3. Please report seeds and mean +/- SD for Tables 1-3, and add a multiplicity correction for S10.
  4. Does the removal of billing codes extend to DRG codes, and can discharge-coded events that postdate the query time leak into the retrieved context?
  5. What is the code/data URL? The current "link" placeholder prevents verification.
4 comments
AuthorsOpenReview ↗

Response to Reviewer xmev comments (1/13), (2/13), (3/13), (4/13), (5/13), and (6/13)

We thank Reviewer xmev for the constructive feedback. We are glad that the reviewer appreciates our attention to data-leakage hygiene.

Comment (1/13):

I have significant concerns about the paper’s soundness that do not appear easy to address.

We have addressed each of the soundness concerns raised:

  • We corrected all reporting inconsistencies.
  • We applied multiplicity correction to our statistical tests.
  • We clarified how prototypes are assigned.
  • We reran experiments with multiple seeds and now report mean ± SD.
  • We clarified that DRG codes were already excluded from the inputs.
  • We replaced the hyperlinked text with the full URL to our code.
  • We clarified the EHRSHOT results. We hope these revisions resolve the reviewer’s concerns about soundness. Each revision is described in more detail below.

Comment (2/13):

The headline numbers cannot be reproduced from the paper's own tuning and ablation tables: Appendix S4 reports Ts = 0.22 for the one-year task, outside the search space {0.1, 0.15, 0.2, 0.25} declared in S3, and S13 shows the best one-year setting at Ts = 0.20 (0.853 AUROC / 0.564 AUPRC).

Comment (3/13):

The in-hospital task is inconsistent the same way: S4 uses 0.20 while S13 reports 0.25 as best (0.950 / 0.736).

Comment (4/13):

Which exact configuration produced the headline 0.750/0.141 and 0.853/0.564 results? Please reconcile S4, S12, and S13, including the Ts = 0.22 value.

We thank the reviewer for raising this inconsistency.

  • We double-checked the best configuration of each task and found that the entries in Table S4 were mistakenly reported while copying them. The correct values of Ts for the 1-Year Mortality and In-Hospital Mortality tasks are 0.2 and 0.25. These have been corrected in Table S4.
  • These corrections agree with results reported in Table S13.
  • We also updated Table S12.

Updates to Table S4: Best configuration selected for 1-year mortality and In-hospital Mortality after hyperparameter optimization.

HyperparameterIn-hospital Mortality1YR Mortality
Chunking StrategyEvent-BasedCare-Stage
Prototype count256512
(Tq,Th)(T_q,T_h)(0.05,0.2)(0.05,0.2)(0.05,0.2)(0.05,0.2)
Weighting temperature (TsT_s)0.250.2
Usage penalty (λu\lambda_u)0.0050.006
Learning rate1.909×10−41.909 \times 10^{-4}1.948×10−51.948 \times 10^{-5}
Weight decay1.380×10−31.380 \times 10^{-3}1.54×10−31.54 \times 10^{-3}

Updated Table S13: Effect of the softmax temperature on model performance.

TsT_s1-Year Mortality AUROC1-Year Mortality AUPRCIn-Hospital Mortality AUROCIn-Hospital Mortality AUPRC
0.100.8450.5400.9470.728
0.150.8520.5520.9480.730
0.200.8530.5640.9480.732
0.250.8510.5530.9500.736

Updated Table S12: Ablation of number of prototypes.

# PrototypesICU Readmission 30d AUROCICU Readmission 30d AUPRC1-Year Mortality AUROC1-Year Mortality AUPRC
640.7500.1410.8500.558
1280.7470.1370.8510.551
2560.7430.1350.8490.554
5120.7390.1290.8530.564

Comment (5/13):

Table 5 reports ICU prototype entropies of 6.198 and 6.232, which exceed the maximum possible for the 64-prototype ICU configuration (ln 64 = 4.159); values near ln 512 = 6.238 suggest a different prototype count was used, and since the ablation never reproduces the headline values, I could not determine which configuration produced them.

Comment (6/13):

What prototype count and model size produced the Table 5 ICU entropies, and can Table 5 be reproduced from a stated configuration?

The ICU entropy entries in Table 5 were left over from an earlier 512-prototype run, which is why they exceeded ln 64. We corrected them and they are consistent with all performance results: the ICU configuration uses 64 prototypes, and the entropies are now 4.149 and 4.155 (maximum ln 64 = 4.159). Corrections are shown below for updated rows in Table 5.

Updated rows in Table 5: Ablation of usage regularization on ICU 30-day readmission.

Usage regularization (λu\lambda_u)Query p MaxQuery EntropyHistory p MaxHistory EntropyAUROCAUPRC
×0.6331.5570.0644.0610.7150.118
✓0.0574.1490.0234.1550.7500.141
AuthorsOpenReview ↗

Response to Reviewer xmev comments (7/13) and (8/13)

Comment (7/13):

On EHRSHOT, the paper reports no confidence intervals and no significance tests, and uses quoted rather than re-run baselines for eight of nine headline tasks;

We thank the reviewer for this comment. We would like to indicate that EHRSHOT [1] is a standardized benchmark published in the NeurIPS Datasets and Benchmarks Track. The benchmark provides fixed patient splits, task definitions, and an evaluation protocol precisely so that results from different groups are directly comparable. To ensure consistency with their baselines, we used their provided config files (official data, splits, and evaluation protocol without modification) and we used their provided pretrained CLMBR checkpoint.

This way we ensured that EHR-RAGp is built on the same pretrained model that the leaderboard reports. The CLMBR baseline results are taken from the benchmark’s public leaderboard, and the long-context results from the Context Clues study [2] (ICLR 2025), a peer-reviewed evaluation of long-context models using the same config.

The purpose of a benchmark is to avoid re-implementation differences, and re-running published models in-house would risk introducing deviations from the reported protocol, not removing them. We therefore report the published numbers and cite their source. We also made a clear statement that these comparisons are descriptive in: “we make no claim of superiority or statistical significance on EHRSHOT”, which the reviewer supports. Our goal was to show how EHR-RAGp compares with models trained on long contexts as a secondary analysis. We agree that this was not clear in the paper so we added a clear statement.

References:

[1] Wornow M, Thapa R, Steinberg E, Fries J, Shah N. Ehrshot: An ehr benchmark for few-shot evaluation of foundation models. Advances in Neural Information Processing Systems. 2023 Dec 15;36:67125-37.

[2] Wornow M, Bedi S, Fuentes Hernandez MA, Steinberg E, Fries J, Ré C, Koyejo S, Shah N. Context clues: evaluating long context models for clinical prediction tasks on EHR data. In International Conference on Learning Representations 2025 May 1 (Vol. 2025, pp. 16518-1)

Comment (8/13):

Section S10 reports p-values from 0.0098 to 0.0355 across roughly 112 tests with no multiplicity correction. Add a multiplicity correction for S10.

We applied Holm correction across all 112 comparisons in Table S10, with controlling the family-wise error rate at 0.05. After correction, we found that 108 out of all 112 comparisons remain statistically significant. The four non-significant comparisons concern EHR-RAGp’s ICU-readmission AUPRC against BigBird, GenHPF, RoBERTa, and RoFormer. We will report the adjusted p-values and update the corresponding significance markers and claims in the manuscript. The correction results are shown below.

Updated Table S10: Adjusted p-values for paired patient-level bootstrap comparisons between EHR-RAGp and each baseline. Adjusted p-values were computed with Holm’s step-down procedure over all 112 comparisons (14 baselines × 8 metrics), using 10,000 bootstrap resamples with the add-one convention p = (k+1)/(B+1). The smallest attainable raw p-value is therefore 1/10,001 ≈ 0.0001, which Holm scales to 112 × 0.0001 = 0.0112; entries equal to 0.0112 are comparisons where no resample favored the baseline, so their true p-value is below this resolution limit.

ModelICU-Readmit 30d AUROCICU-Readmit 30d AUPRCIn-hospital Mortality AUROCIn-hospital Mortality AUPRCLong LOS 7d AUROCLong LOS 7d AUPRC1YR Mortality AUROC1YR Mortality AUPRC
DescEmb I0.01120.01170.01120.01120.01120.01120.01120.0112
DescEmb II0.01120.01120.01120.01120.01120.01120.01120.0112
GenHPF0.01120.23950.01120.01120.01120.01120.01120.0112
REMed0.01120.01120.01120.01120.01120.01120.01120.0112
Med-BERT0.01120.01120.01120.01120.01120.01120.01120.0112
CEHR-BERT0.01120.01120.01120.01120.01120.01120.01120.0112
BEHRT0.01120.01120.01120.01120.01120.01120.01120.0112
Hi-BEHRT0.01120.01170.01120.01120.01120.01120.01120.0112
EHRMamba0.01120.01170.01120.01120.01120.01120.01120.0112
ModernBERT0.01120.01120.01120.01120.01120.01120.01120.0112
Longformer0.01120.01120.01120.01120.01120.01120.01120.0112
BigBird0.04900.07380.01120.01120.01120.01120.01120.0112
RoFormer0.01120.06520.01120.01120.01120.01120.01120.0112
RoBERTa0.01170.07380.01120.01120.01120.01120.01120.0112
AuthorsOpenReview ↗

Response to Reviewer xmev comments (9/13), (10/13), and (11/13)

Comment (9/13):

Table 5 shows prototype assignments operating at about 99% of maximum entropy, meaning the weights are nearly uniform, which sits awkwardly with the claim that prototypes meaningfully guide retrieval.

The entropy in Table 5 is that of the batch-averaged prototype assignment, the quantity our usage regularizer maximizes. Hence, a value near ln K means that all prototypes are used, not that every chunk receives the same weight. The corrected values in Table 5 above show the difference: without the regularizer (×), the assignments collapse (a single prototype takes 63% of the query mass, with entropy 1.557) and performance drops (0.715/0.118); with it (✓), the usage is balanced across the 64 prototypes (entropy 4.149 and 4.155, close to ln 64 = 4.159) and performance improves (0.750/0.141). Near-maximal entropy is therefore the intended healthy outcome, and the failure case is the low-entropy collapse. The chunk weights are a separate quantity, computed from the agreement between the query and chunk assignments. We will clarify in the Table 5 caption that the entropies are computed on the batch-averaged assignments.

Comment (10/13):

No seeds or error bars are reported anywhere, and the prototype-attribution gain (AUROC 0.008-0.015) is smaller than the spread of the prototype-count sweep itself (0.018 in S12), so the benefit of the central mechanism is within run-to-run variation.

Comment (11/13):

Please report seeds and mean +/- SD for Tables 1-3

We ran new experiments and now report mean ± SD over 3 random seeds for Tables 1–3 of the manuscript, with the hyperparameters of each model fixed to its best configuration.

Updated Table 1: Performance on the four MIMIC-IV tasks (test set). AUROC and AUPRC, reported as mean ± SD over 3 random seeds, with the hyperparameters of each model fixed to its best configuration.

ModelICU AUROCICU AUPRCIn-hospital AUROCIn-hospital AUPRCLong LOS AUROCLong LOS AUPRC1-year AUROC1-year AUPRC
DescEmb I0.663±0.0020.663 \pm 0.0020.081±0.0040.081 \pm 0.0040.903±0.0010.903 \pm 0.0010.617±0.0010.617 \pm 0.0010.821±0.0010.821 \pm 0.0010.402±0.0030.402 \pm 0.0030.753±0.0010.753 \pm 0.0010.372±0.0040.372 \pm 0.004
DescEmb II0.609±0.0030.609 \pm 0.0030.080±0.0090.080 \pm 0.0090.923±0.0020.923 \pm 0.0020.656±0.0070.656 \pm 0.0070.839±0.0030.839 \pm 0.0030.443±0.0020.443 \pm 0.0020.806±0.0010.806 \pm 0.0010.472±0.0050.472 \pm 0.005
GenHPF0.697±0.0030.697 \pm 0.0030.126±0.0020.126 \pm 0.0020.912±0.0030.912 \pm 0.0030.641±0.0020.641 \pm 0.0020.843±0.0020.843 \pm 0.0020.455±0.0030.455 \pm 0.0030.782±0.0020.782 \pm 0.0020.418±0.0010.418 \pm 0.001
Med-BERT0.663±0.0070.663 \pm 0.0070.077±0.0040.077 \pm 0.0040.896±0.0040.896 \pm 0.0040.586±0.0080.586 \pm 0.0080.815±0.0020.815 \pm 0.0020.420±0.0080.420 \pm 0.0080.722±0.0100.722 \pm 0.0100.330±0.0120.330 \pm 0.012
REMed0.566±0.0050.566 \pm 0.0050.064±0.0050.064 \pm 0.0050.860±0.0060.860 \pm 0.0060.482±0.0030.482 \pm 0.0030.786±0.0040.786 \pm 0.0040.349±0.0020.349 \pm 0.0020.683±0.0030.683 \pm 0.0030.283±0.0030.283 \pm 0.003
CEHR-BERT0.607±0.0030.607 \pm 0.0030.065±0.0050.065 \pm 0.0050.927±0.0030.927 \pm 0.0030.673±0.0040.673 \pm 0.0040.843±0.0020.843 \pm 0.0020.443±0.0070.443 \pm 0.0070.803±0.0060.803 \pm 0.0060.464±0.0110.464 \pm 0.011
BEHRT0.666±0.0030.666 \pm 0.0030.075±0.0020.075 \pm 0.0020.902±0.0040.902 \pm 0.0040.608±0.0080.608 \pm 0.0080.824±0.0020.824 \pm 0.0020.444±0.0030.444 \pm 0.0030.770±0.0040.770 \pm 0.0040.385±0.0090.385 \pm 0.009
Hi-BEHRT0.568±0.0040.568 \pm 0.0040.064±0.0080.064 \pm 0.0080.884±0.0030.884 \pm 0.0030.536±0.0050.536 \pm 0.0050.801±0.0030.801 \pm 0.0030.400±0.0070.400 \pm 0.0070.704±0.0120.704 \pm 0.0120.299±0.0100.299 \pm 0.010
EHRMamba0.662±0.0030.662 \pm 0.0030.080±0.0020.080 \pm 0.0020.931±0.0030.931 \pm 0.0030.659±0.0080.659 \pm 0.0080.832±0.0040.832 \pm 0.0040.433±0.0080.433 \pm 0.0080.789±0.0170.789 \pm 0.0170.434±0.0050.434 \pm 0.005
ModernBERT0.608±0.0040.608 \pm 0.0040.070±0.0030.070 \pm 0.0030.921±0.0020.921 \pm 0.0020.642±0.0030.642 \pm 0.0030.828±0.0020.828 \pm 0.0020.438±0.0020.438 \pm 0.0020.762±0.0030.762 \pm 0.0030.375±0.0060.375 \pm 0.006
Longformer0.645±0.0020.645 \pm 0.0020.094±0.0010.094 \pm 0.0010.910±0.0020.910 \pm 0.0020.577±0.0010.577 \pm 0.0010.819±0.0030.819 \pm 0.0030.420±0.0020.420 \pm 0.0020.758±0.0030.758 \pm 0.0030.373±0.0040.373 \pm 0.004
BigBird0.714±0.0050.714 \pm 0.0050.117±0.0060.117 \pm 0.0060.915±0.0020.915 \pm 0.0020.645±0.0020.645 \pm 0.0020.828±0.0020.828 \pm 0.0020.440±0.0050.440 \pm 0.0050.790±0.0030.790 \pm 0.0030.428±0.0040.428 \pm 0.004
RoFormer0.696±0.0040.696 \pm 0.0040.119±0.0040.119 \pm 0.0040.928±0.0010.928 \pm 0.0010.675±0.0030.675 \pm 0.0030.847±0.0020.847 \pm 0.0020.467±0.0050.467 \pm 0.0050.808±0.0020.808 \pm 0.0020.467±0.0060.467 \pm 0.006
RoBERTa0.725±0.0030.725 \pm 0.0030.126±0.0020.126 \pm 0.0020.924±0.0010.924 \pm 0.0010.670±0.0020.670 \pm 0.0020.842±0.0010.842 \pm 0.0010.467±0.0020.467 \pm 0.0020.804±0.0010.804 \pm 0.0010.466±0.0030.466 \pm 0.003
EHR-RAGp0.751±0.0020.751 \pm 0.0020.141±0.0020.141 \pm 0.0020.950±0.0020.950 \pm 0.0020.737±0.0020.737 \pm 0.0020.863±0.0030.863 \pm 0.0030.508±0.0040.508 \pm 0.0040.851±0.0020.851 \pm 0.0020.562±0.0020.562 \pm 0.002
AuthorsOpenReview ↗

Response to Reviewer xmev comments (11/13), (12/13), and (13/13)

Comment (11/13): Continued

Updated Table 3: Performance on the four MIMIC-IV tasks (test set) for existing baselines with/without EHR-RAGp. AUROC and AUPRC, reported as mean ± SD over 3 random seeds, with the hyperparameters of each model fixed to its best configuration.

ModelEHR-RAGpICU AUROCICU AUPRCIn-hospital AUROCIn-hospital AUPRCLong LOS AUROCLong LOS AUPRC1-year AUROC1-year AUPRC
Med-BERTx0.663±0.0070.663 \pm 0.0070.077±0.0040.077 \pm 0.0040.896±0.0040.896 \pm 0.0040.586±0.0080.586 \pm 0.0080.815±0.0020.815 \pm 0.0020.420±0.0080.420 \pm 0.0080.722±0.0100.722 \pm 0.0100.330±0.0120.330 \pm 0.012
Med-BERT✓0.712±0.0030.712 \pm 0.0030.114±0.0030.114 \pm 0.0030.937±0.0030.937 \pm 0.0030.698±0.0080.698 \pm 0.0080.842±0.0010.842 \pm 0.0010.452±0.0040.452 \pm 0.0040.820±0.0030.820 \pm 0.0030.471±0.0060.471 \pm 0.006
CEHR-BERTx0.607±0.0030.607 \pm 0.0030.065±0.0050.065 \pm 0.0050.927±0.0030.927 \pm 0.0030.673±0.0040.673 \pm 0.0040.843±0.0020.843 \pm 0.0020.443±0.0070.443 \pm 0.0070.803±0.0060.803 \pm 0.0060.464±0.0110.464 \pm 0.011
CEHR-BERT✓0.694±0.0040.694 \pm 0.0040.103±0.0020.103 \pm 0.0020.940±0.0020.940 \pm 0.0020.711±0.0030.711 \pm 0.0030.847±0.0010.847 \pm 0.0010.454±0.0030.454 \pm 0.0030.821±0.0010.821 \pm 0.0010.492±0.0030.492 \pm 0.003
BEHRTx0.666±0.0030.666 \pm 0.0030.075±0.0020.075 \pm 0.0020.902±0.0040.902 \pm 0.0040.608±0.0080.608 \pm 0.0080.824±0.0020.824 \pm 0.0020.444±0.0030.444 \pm 0.0030.770±0.0040.770 \pm 0.0040.385±0.0090.385 \pm 0.009
BEHRT✓0.722±0.0010.722 \pm 0.0010.123±0.0020.123 \pm 0.0020.941±0.0000.941 \pm 0.0000.709±0.0030.709 \pm 0.0030.850±0.0020.850 \pm 0.0020.461±0.0030.461 \pm 0.0030.821±0.0040.821 \pm 0.0040.486±0.0040.486 \pm 0.004
Hi-BEHRTx0.568±0.0040.568 \pm 0.0040.064±0.0080.064 \pm 0.0080.884±0.0030.884 \pm 0.0030.536±0.0050.536 \pm 0.0050.801±0.0030.801 \pm 0.0030.400±0.0070.400 \pm 0.0070.704±0.0120.704 \pm 0.0120.299±0.0100.299 \pm 0.010
Hi-BEHRT✓0.684±0.0020.684 \pm 0.0020.102±0.0070.102 \pm 0.0070.934±0.0010.934 \pm 0.0010.689±0.0020.689 \pm 0.0020.834±0.0010.834 \pm 0.0010.439±0.0030.439 \pm 0.0030.784±0.0020.784 \pm 0.0020.399±0.0030.399 \pm 0.003

New Supplementary Table: EHRSHOT results for EHR-RAGp with AUROC reported as mean ± SD over 3 random seeds, with the hyperparameters of each model fixed to its best configuration.

ICULLOSRAMILPHLHTCELPC
0.878±0.0030.878 \pm 0.0030.842±0.0030.842 \pm 0.0030.816±0.0010.816 \pm 0.0010.762±0.0010.762 \pm 0.0010.861±0.0020.861 \pm 0.0020.719±0.0020.719 \pm 0.0020.748±0.0010.748 \pm 0.0010.782±0.0030.782 \pm 0.0030.870±0.0030.870 \pm 0.003

Comment (12/13):

Does the removal of billing codes extend to DRG codes, and can discharge-coded events that postdate the query time leak into the retrieved context?

For leakage prevention, we exclude all kinds of billing codes that exist in the dataset from appearing in the query segment that serves as a direct input to our model and all baselines, including DRG codes. We have made this statement more clear in Section B.5 Leakage Prevention as follows:

(3) We examine each query if its contents could include any billing codes that might leak labels. Billing codes accounted for are ICD diagnosis codes, ICD procedure codes, DRG codes, as well as MEDS specific codes (i.e. MEDS_DEATH code). ”

Comment (13/13):

What is the code/data URL? The current "link" placeholder prevents verification.

The anonymized code repository is already linked in the manuscript: the word “link” in the contribution section (page 2) is a clickable hyperlink to https://anonymous.4open.science/r/EHR-RAGp-5857. We have now included the URL in full in the text and in the Reproducibility Statement, so that it is also visible in printed copies. We hope these additions address the reviewer’s concerns. We would be grateful if the reviewer would consider revising their score and we are happy to answer any further questions.

Reviewer_9FsWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces EHR-RAGp, a retrieval-augmented framework for long-horizon structured EHR modeling. The patient timeline is partitioned into multi-granular chunks. A frozen MLM-pretrained EHR encoder selects top-MM candidate chunks via cosine similarity to the current visit's query segment. A learnable set of prototypes assigns soft weights to these candidates using asymmetric temperatures and cross-entropy agreement. The weighted chunks and query are then fused via a shallow transformer for downstream binary prediction (trained via BCE + entropy regularization). Evaluated across 4 MIMIC-IV and 9 EHRSHOT tasks, EHR-RAGp improves over a RoBERTa backbone and enhances four existing EHR models.

Critical strengths
  • Plug-and-Play Generalizability: EHR-RAGp consistently boosts performance across 32 model/task/metric combinations when added to Med-BERT, CEHR-BERT, BEHRT, and Hi-BEHRT (supported by paired bootstrap tests). This strongly validates the benefit of integrating long-horizon history.
  • Rigorous MIMIC-IV Protocol: Re-training baselines on a unified feature set with bootstrap CIs ensures reliable within-paper comparisons.
Critical weaknesses
  1. Unjustified Prototype Reranking Mechanism: The prototype module operates purely as a re-weighting layer over a fixed candidate set without influencing candidate selection. It lacks comparison against natural weighting/reranking baselines: (a) LLM-as-a-reranker over serialized chunks, (b) retriever cosine similarity scores, or (c) uniform weighting relying solely on the fusion transformer's attention. Case studies (Figs S5–S6) reveal that most retrieved chunks originate from the current visit, raising concerns that a trivial recency baseline could achieve similar gains. Zero-shot small LLMs in Table S5 fail to address LLM reranking capabilities. Furthermore, labeling the framework "end-to-end" is an overstatement since gradients do not flow to retrieval selection.
  2. Incomplete Baseline Comparisons:
    • Missing Modern SOTA: Lacks comparisons against generative EHR foundation models like ETHOS (open-source, MIMIC-IV pretrained) and competitive tree models (e.g., Count-based GBM from EHRSHOT).
    • Unclear Protocol Details: EHRSHOT long-context results are directly copied without paired tests, and baseline context budgets (e.g., ModernBERT's 8K capacity) are omitted, making fair comparison unclear.
Questions for authors
  1. Reranking Baseline & Efficiency: Holding the top-MM candidates and backbone fixed, how does prototype weighting perform against:
    • (a) An LLM reranker scoring serialized chunk relevance?
    • (b) Direct query–chunk cosine similarity weighting?
    • (c) Uniform weighting with standard fusion transformer attention?
    • Please provide relative compute/latency costs.
  2. Retrieval Analysis & Recency: How do random and recency-based chunk selection compare to EHR-RAGp? What percentage of retrieved chunks actually originate from prior visits vs. the current visit across the test set?
  3. Ablation Setup: How is "w/o Prototypes" specifically implemented (unweighted fusion, similarity-weighted, or otherwise)?
  4. Additional Baselines: Can you evaluate ETHOS, Count-based GBM, and REMed (in its native setup), while explicitly reporting the exact token/context length assigned to every baseline?
  5. Inconsistencies & Data Discrepancies:
    • Table S11 columns appear mislabeled (In-Hospital Mortality instead of ICU Readmission / 1-year Mortality).
    • Table S12 reports 0.744/0.139 for 64 prototypes on ICU readmission, contradicting Table S4 (selected config) and Table 1 (0.750/0.141).
    • Ts=0.22T_s = 0.22 in Table S4 falls outside the search space listed in Table S3.
    • Are reported main results from a single run or the best across multiple seeds?
  6. Literature (Suggestion): Please cite and discuss EHR-RAG (arXiv:2601.21340), which addresses a very similar topic with identical nomenclature.
9 comments
AuthorsOpenReview ↗

Response to Reviewer 9FsW comments (1/15), (2/15), and (3/15)

We thank Reviewer 9FsW for the careful review. We appreciate that the reviewer recognizes the plug-and-play generalizability of EHR-RAGp: it consistently improves performance in all 32 model/task/metric combinations when added to Med-BERT, CEHR-BERT, BEHRT, and Hi-BEHRT (supported by paired bootstrap tests), which supports the benefit of integrating long-horizon patient history. We also appreciate the reviewer’s recognition of our MIMIC-IV protocol, in which all baselines are re-trained on a unified feature set with bootstrap confidence intervals, ensuring reliable within-paper comparisons.

Comment (1/15)

I have significant concerns about the paper’s soundness that do not appear easy to address.

We have addressed each of the soundness concerns raised:

  1. We added chunk-weighting baselines (self-attention, uniform, cosine similarity) with their compute costs, and a pilot cost study of an LLM reranker.
  2. We clarified the intended objective of the LLM zero-shot results.
  3. We revised the use of the term “end-to-end”.
  4. We clarified the setup of EHRSHOT results.
  5. We ran experiments with matched context budgets, and stated the context budget of every baseline.
  6. We clarified the ablation set up. 7.We fixed some inconsistencies flagged by the reviewer.
  7. We reported mean ± SD over 3 seeds for the main results.
  8. We added random and recency chunk-selection analyses, and the share of retrieved chunks from the current and prior visits.
  9. We added a count-based GBM baseline on MIMIC-IV and EHRSHOT, and clarified that REMed was run in its native form.
  10. We explained why ETHOS could not yet be run, and discussed the related EHR-RAG preprint.

We hope these revisions resolve the reviewer’s concerns about soundness. Each revision is described in more detail below.

Comment (2/15)

Zero-shot small LLMs in Table S5 fail to address LLM reranking capabilities.

We thank the reviewer for this comment. Table S5 was not intended to evaluate LLM reranking. It reports zero-shot LLMs used as direct outcome predictors, as supplementary context, and these models are not part of any comparison in the main paper. The protocol in the appendix describes them as predictors, and we make no claim about their reranking ability. We address LLM reranking directly in our response to your other reranking-baseline question, where an LLM scores the relevance of each retrieved chunk under a protocol of its own. We will also state this explicitly in the updated manuscript to clarify the purpose of Table S5.

Comment (3/15)

Furthermore, labeling the framework "end-to-end" is an overstatement since gradients do not flow to retrieval selection.

We clearly state in Figure 1 and Section 3 that the retriever R is frozen and that the search indices are precomputed apriori. By “end-to-end” we mean that the backbone encoder, the prototype module and the fusion module are trained jointly from the task loss, with gradients flowing through all of them. Keeping the retriever fixed is a deliberate design choice, since it allows a precomputed index and constant-cost inference. To remove any ambiguity, we will describe the stage as “end-to-end fine-tuning with a frozen retriever” throughout the paper.

AuthorsOpenReview ↗

Response to Reviewer 9FsW comments (4/15) and (5/15)

Comment (4/15)

Unclear Protocol Details: EHRSHOT long-context results are directly copied without paired tests,

We thank the reviewer for this comment. We would like to indicate that EHRSHOT [1] is a standardized benchmark published in the NeurIPS Datasets and Benchmarks Track. The benchmark provides fixed patient splits, task definitions, and an evaluation protocol precisely so that results from different groups are directly comparable. To ensure consistency with their baselines, we used their provided config files (official data, splits, and evaluation protocol without modification) and we used their provided pretrained CLMBR checkpoint.

This way we ensured that EHR-RAGp is built on the same pretrained model that the leaderboard reports. The CLMBR baseline results are taken from the benchmark’s public leaderboard, and the long-context results from the Context Clues study [2] (ICLR 2025), a peer-reviewed evaluation of long-context models using the same config.

The purpose of a benchmark is to avoid re-implementation differences, and re-running published models in-house would risk introducing deviations from the reported protocol. We therefore report the published numbers and cite their source. We also made a clear statement that these comparisons are descriptive in: “we make no claim of superiority or statistical significance on EHRSHOT”. Our goal was to show how EHR-RAGp compares with models trained on long contexts as a secondary analysis. We agree that this was not clear in the paper, so we added a clear statement.

References

[1] Wornow M, Thapa R, Steinberg E, Fries J, Shah N. Ehrshot: An ehr benchmark for few-shot evaluation of foundation models. Advances in Neural Information Processing Systems. 2023 Dec 15;36:67125-37.

[2] Wornow M, Bedi S, Fuentes Hernandez MA, Steinberg E, Fries J, Ré C, Koyejo S, Shah N. Context clues: evaluating long context models for clinical prediction tasks on EHR data. In International Conference on Learning Representations 2025 May 1 (Vol. 2025, pp. 16518-1)

Comment (5/15)

Baseline context budgets (e.g., ModernBERT's 8K capacity) are omitted, making fair comparison unclear.

We agree that the baselines’ context budgets were not stated clearly. The budgets are 512 for DescEmb, GenHPF, Med-BERT, CEHR-BERT and BEHRT; 1,024 for Hi-BEHRT, Longformer, BigBird and RoFormer; 2,048 for EHRMamba; and 4,096 for ModernBERT, whose native capacity is 8K. REMed reads the full history and selects 128 events from it. EHR-RAGp uses the same 512-event query segment as the 512-token baselines, and adds retrieved history of up to 24 chunks of 256 tokens from earlier in the timeline. The comparison therefore contrasts two ways of using more than the most recent window: reading a longer contiguous window (the 1,024 to 4,096 baselines) against retrieving the relevant older segments.

We conducted a new analysis to assess performance with a matched budget (Table 1 below), with EHR-RAGp and ModernBERT as a long-context baseline, at the same context budget of 6,656 tokens (EHR-RAGp’s maximum: 512 query + 24 × 256 retrieved tokens). We chose ModernBERT because it’s the only baseline that can handle up to 8K tokens at once. Both models run on the same GPU (A100 80GB) with BF16 mixed precision, the SGD optimizer, up to 75 epochs, with each model at its best hyperparameters. Batch size of 12 for ModernBERT is the maximum that can fit in the GPU memory, whereas batch size of 24 for EHR-RAGp is the same used during training. Overall, EHR-RAGp achieves a better performance. The new Table (1) will be added to the main results section and the new Table (2) with detailed per task results will be added to the appendix. We will also add the context budgets of all models as supplementary information.

Table (1): Controlled comparisons under matched context length, parameter budget and computational setting.

ModelInput tokensMax batch sizeParams (M)GPUPrecisionOptimizerMax EpochsTrain time per epoch (min)Mean Runtime (h)Inference latency (ms/example)Peak GPU Mem. (Training) (GB)Peak GPU Mem. (Inference, per example) (GB)Mean AUROC (macro)Mean AUPRC (macro)
ModernBERT6,65612148A100 80GBBF16 mixedSGD7521346.2528174.61.4600.7860.384
EHR-RAGp6,65624137A100 80GBBF16 mixedSGD753813.48365.40.8810.8550.488
AuthorsOpenReview ↗

Response to Reviewer 9FsW comments (5/15), (6/15), (7/15), and (8/15)

Comment (5/15): Continued

  • Parameter budget: EHR-RAGp has 137M effective parameters (fusion module and prototypes included) compared to 148M for ModernBERT encoder, excluding the default text embedding layer. The EHR embedding Layer is identical for both models, which account for 37M parameters summed to the earlier mentioned effective parameters.

  • Computational settings: EHR-RAGp is much cheaper. It trains 5.6× faster per epoch (38 vs. 213 min) with a mean total runtime of 13.4 vs. 46.25 hours, runs inference 3.4× faster (83 vs. 281 ms per example), and uses less memory in both training (65.4 vs. 74.6 GB, at twice the batch size) and inference (0.881 vs. 1.46 GiB). It is worth noting that building the retrieval index (chunking and encoding each stay’s history with the frozen retriever) takes roughly 57 ms per stay on an A100 a one-time offline cost that is not included in the reported 83 ms inference latency.

  • Performance: At this matched budget, EHR-RAGp reaches a macro-average AUROC/AUPRC of 0.855/0.488, against 0.786/0.384 for ModernBERT.

Table (2): Per-task performance comparison between EHR-RAGp and ModernBERT at matched budgets across all tasks.

ModelInput tokensICU Readmission AUROCICU Readmission AUPRCIn-Hospital Mortality AUROCIn-Hospital Mortality AUPRCLong LOS AUROCLong LOS AUPRC1-Year Mortality AUROC1-Year Mortality AUPRC
ModernBERT6,6560.6380.0710.9210.6590.8270.4370.7590.369
EHR-RAGp6,6560.7500.1410.9500.7360.8670.5090.8530.564

Comment (6/15)

Ablation Setup: How is "w/o Prototypes" specifically implemented (unweighted fusion, similarity-weighted, or otherwise)?

“w/o Prototypes” is self-attention. The fusion transformer’s own self-attention is the only thing that weighs the chunks.

Comment (7/15)

Table S11 columns appear mislabeled (In-Hospital Mortality instead of ICU Readmission / 1-year Mortality).

Thanks for spotting this. The results shown in Table S11, second task, are intended for In-Hospital Mortality, not 1-Year Mortality. We fixed this by updating the column names.

Comment (8/15)

Table S12 reports 0.744/0.139 for 64 prototypes on ICU readmission, contradicting Table S4 (selected config) and Table 1 (0.750/0.141). in Table S4 falls outside the search space listed in Table S3.

We thank the reviewer for raising this inconsistency.

  • We double-checked the best configuration of each task and found that the entries in Table S4 were mistakenly reported while copying them. The correct values of Ts for the 1-Year Mortality and In-Hospital Mortality tasks are 0.2 and 0.25. These have been corrected in Table S4.

  • These corrections agree with results reported in Table S13.

  • We also updated Table S12.

Updates to Table S4: Best configuration selected for 1-year mortality and In-hospital Mortality after hyperparameter optimization.

HyperparameterIn-hospital Mortality1YR Mortality
Chunking StrategyEvent-BasedCare-Stage
Prototype count256512
(Tq,Th)(T_q,T_h)(0.05,0.2)(0.05,0.2)(0.05,0.2)(0.05,0.2)
Weighting temperature (TsT_s)0.250.2
Usage penalty (λu\lambda_u)0.0050.006
Learning rate1.909×10−41.909 \times 10^{-4}1.948×10−51.948 \times 10^{-5}
Weight decay1.380×10−31.380 \times 10^{-3}1.54×10−31.54 \times 10^{-3}

Updated Table S13: Effect of the softmax temperature on model performance.

TsT_s1-Year Mortality AUROC1-Year Mortality AUPRCIn-Hospital Mortality AUROCIn-Hospital Mortality AUPRC
0.100.8450.5400.9470.728
0.150.8520.5520.9480.730
0.200.8530.5640.9480.732
0.250.8510.5530.9500.736

Updated Table S12: Ablation of the number of prototypes.

# PrototypesICU Readmission 30d AUROCICU Readmission 30d AUPRC1-Year Mortality AUROC1-Year Mortality AUPRC
640.7500.1410.8500.558
1280.7470.1370.8510.551
2560.7430.1350.8490.554
5120.7390.1290.8530.564
Reviewer_sH3KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce a retrieval-based framework to integrate the most relevant patient history. It first estimates the relevance of retrieved historical chunks and uses a prototype-guided retrieval module that acts as an alignment mechanism and uses the features for EHR prediction.

Critical strengths

The research is quite interesting, as longitudinal EHRs are irregular, heterogeneous, and can exceed model context limits. Retrieving historical information specific to each patient before using it for prediction is computation efficient and useful.

Critical weaknesses

For me, it is not clear if the performance gain is from the proposed mechanism or from other factors. e.g., Table 4 shows that much of the improvement comes from introducing retrieval alone, while prototype-guided alignment provides relatively modest additional gains.

The fundamental novelty is not clear. The proposed framework mainly combines prototype-based representation learning, similarity-based weighting, and transformer fusion. Some related work, e.g., REMed, did similar things. The cross-entropy alignment is equivalent to KL-divergence-based weighting after softmax normalization.

The comparison is not fair enough to validate the effectiveness of the proposal. EHR-RAGp accesses additional historical information and introduces an extra fusion module, whereas many baselines operate on restricted input sequences.

Questions for authors
  1. Could the authors compare it with simpler alternatives, such as standard attention, cosine-similarity weighting, and uniform aggregation?

  2. Could the authors provide comparisons under matched context lengths, parameter budgets, and computational settings?

  3. Could the authors report training and inference costs, memory consumption, and scalability with increasing history length?

4 comments
AuthorsOpenReview ↗

Response to Reviewer sH3K comments (1/6) and (2/6)

We would like to thank Reviewer sH3K for the careful review and constructive feedback. We appreciate that the reviewer finds our submitted paper (i) interesting (ii) with minor and fixable concerns (iii) and targets the challenge of longitudinal, irregular, and heterogeneous EHR data that can exceed the context limits of existing models. We also appreciate that the reviewer recognizes that retrieving patient-specific history for a prediction is computationally efficient and useful, which is the main motivation behind EHR-RAGp. In summary, we conducted new experiments and added matched-budget comparison, weighting ablation, cost/scalability analysis.

Comment (1/6):

For me, it is not clear if the performance gain is from the proposed mechanism or from other factors. e.g., Table 4 shows that much of the improvement comes from introducing retrieval alone, while prototype-guided alignment provides relatively modest additional gains.

We agree that retrieving patient-specific history accounts for most of the improvement, and this is a central contribution of EHR-RAGp so it is not a weakness. We present EHR-RAGp as an integrated system, with prototype refinement on top of retrieval. Table 4 shows that retrieval provides the largest gains, while the full model achieves a further statistically significant improvement over the ablation without prototypes (p<0.05 for both tasks and across AUROC/AUPRC). These results support both the value of patient-specific retrieval and the additional benefit of incorporating prototypes. We now have two new experiments that address this more: the matched-budget comparison and the weighting ablation (described in more detail to specific questions below).

Comment (2/6):

The fundamental novelty is not clear. The proposed framework mainly combines prototype-based representation learning, similarity-based weighting, and transformer fusion. Some related work, e.g., REMed, did similar things. The cross-entropy alignment is equivalent to KL-divergence-based weighting after softmax normalization.

Although we agree that individual components such as retrieval, chunking, and prototype-based weighting have been explored independently in other domains, our contribution is not the introduction of these components in isolation, but rather their integration into a patient-specific retrieval framework for structured longitudinal EHR modeling, where retrieval must operate over heterogeneous, irregular, and temporally evolving clinical event sequences.

REMed is the closest approach to our work, and is included in our list of baselines. However, REMed scores individual events sampled randomly (128 events max) in a text-based setup, while EHR-RAGp retrieves query-conditioned history segments from code-based timelines exploring four chunking strategies. Furthermore, EHR-RAGp significantly improves over REMed as indicated in Table 1 and Table S10. Summary (Ours vs REMed):

  • ICU readmission 0.750 vs 0.561 AUROC and 0.141 vs 0.060 AUPRC
  • In-hospital mortality 0.950 vs 0.865 AUROC and 0.736 vs 0.501 AURPC
  • Long LoS 7d 0.867 vs 0.786 AUROC and 0.509 vs 0.348 AUPRC
  • 1-Year mortality 0.853 vs 0.680 AUROC and 0.564 vs 0.280 AUPRC

We also agree that the cross-entropy alignment is equivalent to KL-divergence weighting after softmax normalization, since the query entropy is constant across chunks and cancels out. This intends to learn a prototype space with asymmetric query/history temperatures and a usage regularizer that prevents prototype collapse. We have noted this in the updated manuscript.

AuthorsOpenReview ↗

Response to Reviewer sH3K comments (3/6) and (4/6)

Comment (3/6):

The comparison is not fair enough to validate the effectiveness of the proposal. EHR-RAGp accesses additional historical information and introduces an extra fusion module, whereas many baselines operate on restricted input sequences.

Comment (4/6):

Could the authors provide comparisons under matched context lengths, parameter budgets, and computational settings?

We acknowledge that EHR-RAGp accesses historical information and introduces an extra fusion module. We believe that this is an advantage of our proposed work, which is designed to retrieve historical information from the patient's own history. This is opposed to existing clinical baselines, which handle fixed clinical context and rely on feature selection or regular sampling to manage the context. Hence, our proposed work presents itself as a solution to these limitations and allows for extended information usage.

First, the results on the EHRSHOT benchmark compare EHR-RAGp with long-context baselines (4k - 16k), where EHR-RAGp achieves comparable or better performance.

Second, we conducted a new analysis (Table 1 below) to compare EHR-RAGp to ModernBERT as a long-context baseline, at the same context budget of 6,656 tokens (EHR-RAGp’s maximum: 512 query + 24 × 256 retrieved tokens). We chose ModernBERT because it’s the only baseline that can handle up to 8192 tokens at once. Both models run on the same GPU (A100 80GB) with BF16 mixed precision, the SGD optimizer, up to 75 epochs, with each model at its best hyperparameters. Batch size of 12 for ModernBERT is the maximum that can fit in the GPU memory, whereas batch size of 24 for EHR-RAGp is the same used during training. Overall, EHR-RAGp achieves a better performance. The new Table (1) will be added to the main results section and the new Table (2) with detailed per task results will be added to the appendix.

Table (1): Controlled comparisons under matched context length, parameter budget and computational setting.

ModelInput tokensMax batch sizeParams (M)GPUPrecisionOptimizerMax EpochsTrain time per epoch (min)Mean Runtime (h)Inference latency (ms/example)Peak GPU Mem. (Training) (GB)Peak GPU Mem. (Inference, per example) (GB)Mean AUROC (macro)Mean AUPRC (macro)
ModernBERT6,65612148A100 80GBBF16 mixedSGD7521346.2528174.61.4600.7860.384
EHR-RAGp6,65624137A100 80GBBF16 mixedSGD753813.48365.40.8810.8550.488
  • Parameter budget: EHR-RAGp has 137M effective parameters (fusion module and prototypes included) compared to 148M for ModernBERT encoder, excluding the default text embedding layer. The EHR embedding Layer is identical for both models, which account for 37M parameters summed to the earlier mentioned effective parameters.

  • Computational settings: EHR-RAGp is much cheaper. It trains 5.6× faster per epoch (38 vs. 213 min) with a mean total runtime of 13.4 vs. 46.25 hours, runs inference 3.4× faster (83 vs. 281 ms per example), and uses less memory in both training (65.4 vs. 74.6 GB, at twice the batch size) and inference (0.881 vs. 1.46 GiB). It is worth noting that building the retrieval index (chunking and encoding each stay’s history with the frozen retriever) takes roughly 57 ms per stay on an A100 a one-time offline cost that is not included in the reported 83 ms inference latency.

  • Performance: At this matched budget, EHR-RAGp reaches a macro-average AUROC/AUPRC of 0.855/0.488, against 0.786/0.384 for ModernBERT.

Table (2): Per-task performance comparison between EHR-RAGp and ModernBERT at matched budgets across all tasks.

ModelInput tokensICU Readmission AUROCICU Readmission AUPRCIn-Hospital Mortality AUROCIn-Hospital Mortality AUPRCLong LOS AUROCLong LOS AUPRC1-Year Mortality AUROC1-Year Mortality AUPRC
ModernBERT6,6560.6380.0710.9210.6590.8270.4370.7590.369
EHR-RAGp6,6560.7500.1410.9500.7360.8670.5090.8530.564
AuthorsOpenReview ↗

Response to Reviewer sH3K comments (5/6) and (6/6)

Comment (5/6):

Could the authors compare it with simpler alternatives, such as standard attention, cosine-similarity weighting, and uniform aggregation?

We thank the reviewer for this suggestion. We conducted new analysis accordingly (see new Table 3 below). All variants use the same retrieved top-M chunks, backbone, fusion transformer, and differ only in how the chunks are weighted:

  • Standard self-attention: This is the “w/o Prototypes” setting mentioned in the original Table 4 in the submitted manuscript.
  • Uniform aggregation: Every valid chunk gets equal weight of 1/M.
  • Cosine-similarity weighting: A softmax over the retriever’s query–chunk cosine scores, with the temperature tuned over the same grid as EHR-RAGp.
  • EHR-RAGp: Prototype weighting mechanism.

Results are mean ± SD over 3 seeds on all four tasks. Prototype weighting gives the best average AUROC/AUPRC (0.854/0.487), against 0.844/0.463 for attention only, 0.833/0.447 for uniform and 0.839/0.463 for cosine similarity. The learned prototype weighting achieves the best performance in most tasks and metrics, and a comparable performance to cosine similarity in long length of stay. No single simple alternative is consistently second. These results will be added to the ablations section (extension of the original Table 4).

Table (3): Comparison of chunk-weighting strategies (mean ± SD over 3 seeds).

WeightingICU Readmission AUROCICU Readmission AUPRCIn-Hospital Mortality AUROCIn-Hospital Mortality AUPRCLong LOS AUROCLong LOS AUPRC1-Year Mortality AUROC1-Year Mortality AUPRCMean AUROCMean AUPRC
Self-attention0.735±0.0010.735 \pm 0.0010.135±0.0020.135 \pm 0.0020.941±0.0020.941 \pm 0.0020.713±0.0030.713 \pm 0.0030.857±0.0020.857 \pm 0.0020.486±0.0050.486 \pm 0.0050.842±0.0020.842 \pm 0.0020.517±0.0030.517 \pm 0.0030.8440.8440.4630.463
Uniform (1/M1/M)0.706±0.0040.706 \pm 0.0040.120±0.0020.120 \pm 0.0020.938±0.0020.938 \pm 0.0020.714±0.0020.714 \pm 0.0020.848±0.0030.848 \pm 0.0030.469±0.0040.469 \pm 0.0040.838±0.0030.838 \pm 0.0030.486±0.0040.486 \pm 0.0040.8330.8330.4470.447
Cosine similarity0.716±0.0020.716 \pm 0.0020.123±0.0010.123 \pm 0.0010.944±0.0010.944 \pm 0.0010.717±0.0030.717 \pm 0.0030.864±0.0030.864 \pm 0.0030.501±0.0020.501 \pm 0.0020.833±0.0020.833 \pm 0.0020.510±0.0020.510 \pm 0.0020.8390.8390.4630.463
Prototype (EHR-RAGp)0.751±0.0020.751 \pm 0.0020.141±0.0020.141 \pm 0.0020.950±0.0020.950 \pm 0.0020.737±0.0020.737 \pm 0.0020.863±0.0030.863 \pm 0.0030.508±0.0040.508 \pm 0.0040.851±0.0020.851 \pm 0.0020.562±0.0020.562 \pm 0.0020.8540.8540.4870.487

Comment (6/6):

Could the authors report training and inference costs, memory consumption, and scalability with increasing history length? We thank the reviewer for this question.

Training and inference costs (Table 1). On an A100 80GB with BF16, EHR-RAGp trains at 38 min per epoch (mean total runtime 13.4 hours per task), using 65.4 GB of GPU memory at batch size 24. At inference, it takes 83 ms and 0.88 GiB per example at batch size 1.

Scalability (Table 4). We grouped test stays by the number of chunks in their history (from ≤24 to >100) and measured each stage of inference. The forward pass (about 83 ms) and peak memory (0.88 GiB) stay constant, because the model always encodes a fixed maximum number of retrieved chunks (M=24) regardless of the number of chunks available. FAISS search grows only from 0.03 to 0.08 ms, and online chunking, the only component that scales with history, from 0.4 to 13.7 ms. These results will be added to the appendix of the main paper.

Table (4): Scalability with increasing history length.

History group (chunks)FAISS search latency (ms)Online chunking latency (ms)Forward latency (ms/example)Peak GPU inference memory (GiB)
≤240.028 ± 0.0030.368 ± 0.22583.491 ± 0.5450.881
25–500.035 ± 0.0061.112 ± 0.63883.747 ± 0.5680.882
51–1000.046 ± 0.0093.132 ± 1.64083.652 ± 0.5320.882
>1000.075 ± 0.02113.675 ± 17.78283.107 ± 0.7990.881

We hope these additions address the reviewer’s concerns. We would be grateful if the reviewer would consider revising their score and we are happy to answer any further questions.

Dear authors, I appreciate your response to my questions as well as the newly added experiment. I agree that this is an interesting paper but I feel that there is a lot of content and experiments that need to be clarified and revised. I therefore keep my original rating and suggest that you carefully revise your paper for future submission.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal and updates (Submission #9518)

Dear PC, SAC, AC, and Reviewers,

We sincerely thank the reviewers for their constructive feedback, which has strengthened the manuscript. We are encouraged that the reviewers found the problem important and the approach useful: retrieving patient-specific history for long, irregular and heterogeneous EHRs. They also recognized EHR-RAGp’s consistent gains across 32 model/task/metric combinations when added to existing EHR models, the rigorous MIMIC-IV protocol, and our attention to data-leakage prevention.

Based on your feedback, we have made the following updates to the manuscript:

  • Matched budget comparison: We added a matched-budget comparison with ModernBERT at the same context length (6,656 tokens), GPU and precision, together with training and inference costs. We also stated the context budget of every baseline and added a scalability analysis with increasing history length. We also clarified the breadth of baseline chosen for comparison.

  • Statistical reporting: We now report mean ± SD over 3 seeds for the main results (Tables 1–3 and EHRSHOT), and applied Holm correction across all 112 significance tests, where 108 comparisons remain significant.

  • Weighting analysis: We added comparisons of prototype weighting against various common sense weighting mechanisms including: self-attention, uniform and cosine-similarity weighting, with compute and latency costs as requested by the reviewers.

  • Retrieval analysis: We added random and recency chunk-selection comparisons in the same pipeline, and the proportions of retrieved chunks coming from prior versus current visits for each chunking strategy and prediction window.

  • Corrections and clarifications: We corrected the reporting inconsistencies in Tables S4, S11, S12 and Table 5, clarified how prototype entropy is computed, and stated that DRG codes are excluded from the inputs. We also clarified the use of “end-to-end”, the purpose of the zero-shot LLM results, the EHRSHOT protocol, and added the full code URL.

  • Additional baselines: We added a count-based GBM on MIMIC-IV and EHRSHOT, and clarified that REMed was run in its native form. We explain why ETHOS could not yet be evaluated on our cohort: the public weights rest on a different patient split, data version and tokenization, which could introduce leakage. ETHOS-ARES does not release weights. We discuss ETHOS and the EHR-RAG preprint in the related work.

All changes are incorporated in the updated manuscript and described in detail in our point-by-point responses. We are pleased that Reviewer xmev has confirmed they read the response and has no further questions. We remain available to address any further questions/concerns from the other reviewers and the AC.

Thank you again for your time and dedication in reviewing our submission.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.