- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper recasts long-horizon EHR modeling as retrieval. A patient's history is chunked along four granularities (event, time, visit, care stage), embedded with a RoBERTa encoder, stored in a FAISS index, and top-M chunks are retrieved by cosine similarity. A prototype-guided alignment operator soft-reweights chunks by agreement between query and chunk prototype assignments, and a two-layer transformer fuses the result. Evaluation covers four MIMIC-IV ICU tasks and nine EHRSHOT tasks with a CLMBR backbone, claiming improvement in every backbone/task/metric cell tested.
Critical strengths
Leakage hygiene is better than typical here: pretraining uses only the training split, splits are patient-disjoint, normalization is fit on training data only, and the paper explicitly declines significance claims against EHRSHOT long-context baselines it cannot match. Confidence intervals are reported for all MIMIC numbers.
Critical weaknesses
- The headline numbers cannot be reproduced from the paper's own tuning and ablation tables: Appendix S4 reports Ts = 0.22 for the one-year task, outside the search space {0.1, 0.15, 0.2, 0.25} declared in S3, and S13 shows the best one-year setting at Ts = 0.20 (0.853 AUROC / 0.564 AUPRC).
- The in-hospital task is inconsistent the same way: S4 uses 0.20 while S13 reports 0.25 as best (0.950 / 0.736).
- Table 5 reports ICU prototype entropies of 6.198 and 6.232, which exceed the maximum possible for the 64-prototype ICU configuration (ln 64 = 4.159); values near ln 512 = 6.238 suggest a different prototype count was used, and since the ablation never reproduces the headline values I could not determine which configuration produced them.
- No seeds or error bars are reported anywhere, and the prototype-attribution gain (AUROC 0.008-0.015) is smaller than the spread of the prototype-count sweep itself (0.018 in S12), so the benefit of the central mechanism is within run-to-run variation.
- On EHRSHOT the paper reports no confidence intervals and no significance tests, and uses quoted rather than re-run baselines for eight of nine headline tasks; Section S10 reports p-values from 0.0098 to 0.0355 across roughly 112 tests with no multiplicity correction.
- Table 5 shows prototype assignments operating at about 99% of maximum entropy, meaning the weights are nearly uniform, which sits awkwardly with the claim that prototypes meaningfully guide retrieval.
Questions for authors
- Which exact configuration produced the headline 0.750/0.141 and 0.853/0.564 results? Please reconcile S4, S12 and S13, including the Ts = 0.22 value.
- What prototype count and model size produced the Table 5 ICU entropies, and can Table 5 be reproduced from a stated configuration?
- Please report seeds and mean +/- SD for Tables 1-3, and add a multiplicity correction for S10.
- Does the removal of billing codes extend to DRG codes, and can discharge-coded events that postdate the query time leak into the retrieved context?
- What is the code/data URL? The current "link" placeholder prevents verification.
4 comments
Response to Reviewer xmev comments (7/13) and (8/13)
Comment (7/13):
On EHRSHOT, the paper reports no confidence intervals and no significance tests, and uses quoted rather than re-run baselines for eight of nine headline tasks;
We thank the reviewer for this comment. We would like to indicate that EHRSHOT [1] is a standardized benchmark published in the NeurIPS Datasets and Benchmarks Track. The benchmark provides fixed patient splits, task definitions, and an evaluation protocol precisely so that results from different groups are directly comparable. To ensure consistency with their baselines, we used their provided config files (official data, splits, and evaluation protocol without modification) and we used their provided pretrained CLMBR checkpoint.
This way we ensured that EHR-RAGp is built on the same pretrained model that the leaderboard reports. The CLMBR baseline results are taken from the benchmark’s public leaderboard, and the long-context results from the Context Clues study [2] (ICLR 2025), a peer-reviewed evaluation of long-context models using the same config.
The purpose of a benchmark is to avoid re-implementation differences, and re-running published models in-house would risk introducing deviations from the reported protocol, not removing them. We therefore report the published numbers and cite their source. We also made a clear statement that these comparisons are descriptive in: “we make no claim of superiority or statistical significance on EHRSHOT”, which the reviewer supports. Our goal was to show how EHR-RAGp compares with models trained on long contexts as a secondary analysis. We agree that this was not clear in the paper so we added a clear statement.
References:
[1] Wornow M, Thapa R, Steinberg E, Fries J, Shah N. Ehrshot: An ehr benchmark for few-shot evaluation of foundation models. Advances in Neural Information Processing Systems. 2023 Dec 15;36:67125-37.
[2] Wornow M, Bedi S, Fuentes Hernandez MA, Steinberg E, Fries J, Ré C, Koyejo S, Shah N. Context clues: evaluating long context models for clinical prediction tasks on EHR data. In International Conference on Learning Representations 2025 May 1 (Vol. 2025, pp. 16518-1)
Comment (8/13):
Section S10 reports p-values from 0.0098 to 0.0355 across roughly 112 tests with no multiplicity correction. Add a multiplicity correction for S10.
We applied Holm correction across all 112 comparisons in Table S10, with controlling the family-wise error rate at 0.05. After correction, we found that 108 out of all 112 comparisons remain statistically significant. The four non-significant comparisons concern EHR-RAGp’s ICU-readmission AUPRC against BigBird, GenHPF, RoBERTa, and RoFormer. We will report the adjusted p-values and update the corresponding significance markers and claims in the manuscript. The correction results are shown below.
Updated Table S10: Adjusted p-values for paired patient-level bootstrap comparisons between EHR-RAGp and each baseline. Adjusted p-values were computed with Holm’s step-down procedure over all 112 comparisons (14 baselines × 8 metrics), using 10,000 bootstrap resamples with the add-one convention p = (k+1)/(B+1). The smallest attainable raw p-value is therefore 1/10,001 ≈ 0.0001, which Holm scales to 112 × 0.0001 = 0.0112; entries equal to 0.0112 are comparisons where no resample favored the baseline, so their true p-value is below this resolution limit.
| Model | ICU-Readmit 30d AUROC | ICU-Readmit 30d AUPRC | In-hospital Mortality AUROC | In-hospital Mortality AUPRC | Long LOS 7d AUROC | Long LOS 7d AUPRC | 1YR Mortality AUROC | 1YR Mortality AUPRC |
|---|---|---|---|---|---|---|---|---|
| DescEmb I | 0.0112 | 0.0117 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 |
| DescEmb II | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 |
| GenHPF | 0.0112 | 0.2395 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 |
| REMed | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 |
| Med-BERT | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 |
| CEHR-BERT | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 |
| BEHRT | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 |
| Hi-BEHRT | 0.0112 | 0.0117 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 |
| EHRMamba | 0.0112 | 0.0117 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 |
| ModernBERT | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 |
| Longformer | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 |
| BigBird | 0.0490 | 0.0738 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 |
| RoFormer | 0.0112 | 0.0652 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 |
| RoBERTa | 0.0117 | 0.0738 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 | 0.0112 |
Response to Reviewer xmev comments (9/13), (10/13), and (11/13)
Comment (9/13):
Table 5 shows prototype assignments operating at about 99% of maximum entropy, meaning the weights are nearly uniform, which sits awkwardly with the claim that prototypes meaningfully guide retrieval.
The entropy in Table 5 is that of the batch-averaged prototype assignment, the quantity our usage regularizer maximizes. Hence, a value near ln K means that all prototypes are used, not that every chunk receives the same weight. The corrected values in Table 5 above show the difference: without the regularizer (×), the assignments collapse (a single prototype takes 63% of the query mass, with entropy 1.557) and performance drops (0.715/0.118); with it (✓), the usage is balanced across the 64 prototypes (entropy 4.149 and 4.155, close to ln 64 = 4.159) and performance improves (0.750/0.141). Near-maximal entropy is therefore the intended healthy outcome, and the failure case is the low-entropy collapse. The chunk weights are a separate quantity, computed from the agreement between the query and chunk assignments. We will clarify in the Table 5 caption that the entropies are computed on the batch-averaged assignments.
Comment (10/13):
No seeds or error bars are reported anywhere, and the prototype-attribution gain (AUROC 0.008-0.015) is smaller than the spread of the prototype-count sweep itself (0.018 in S12), so the benefit of the central mechanism is within run-to-run variation.
Comment (11/13):
Please report seeds and mean +/- SD for Tables 1-3
We ran new experiments and now report mean ± SD over 3 random seeds for Tables 1–3 of the manuscript, with the hyperparameters of each model fixed to its best configuration.
Updated Table 1: Performance on the four MIMIC-IV tasks (test set). AUROC and AUPRC, reported as mean ± SD over 3 random seeds, with the hyperparameters of each model fixed to its best configuration.
| Model | ICU AUROC | ICU AUPRC | In-hospital AUROC | In-hospital AUPRC | Long LOS AUROC | Long LOS AUPRC | 1-year AUROC | 1-year AUPRC |
|---|---|---|---|---|---|---|---|---|
| DescEmb I | ||||||||
| DescEmb II | ||||||||
| GenHPF | ||||||||
| Med-BERT | ||||||||
| REMed | ||||||||
| CEHR-BERT | ||||||||
| BEHRT | ||||||||
| Hi-BEHRT | ||||||||
| EHRMamba | ||||||||
| ModernBERT | ||||||||
| Longformer | ||||||||
| BigBird | ||||||||
| RoFormer | ||||||||
| RoBERTa | ||||||||
| EHR-RAGp |
Response to Reviewer xmev comments (11/13), (12/13), and (13/13)
Comment (11/13): Continued
Updated Table 3: Performance on the four MIMIC-IV tasks (test set) for existing baselines with/without EHR-RAGp. AUROC and AUPRC, reported as mean ± SD over 3 random seeds, with the hyperparameters of each model fixed to its best configuration.
| Model | EHR-RAGp | ICU AUROC | ICU AUPRC | In-hospital AUROC | In-hospital AUPRC | Long LOS AUROC | Long LOS AUPRC | 1-year AUROC | 1-year AUPRC |
|---|---|---|---|---|---|---|---|---|---|
| Med-BERT | x | ||||||||
| Med-BERT | ✓ | ||||||||
| CEHR-BERT | x | ||||||||
| CEHR-BERT | ✓ | ||||||||
| BEHRT | x | ||||||||
| BEHRT | ✓ | ||||||||
| Hi-BEHRT | x | ||||||||
| Hi-BEHRT | ✓ |
New Supplementary Table: EHRSHOT results for EHR-RAGp with AUROC reported as mean ± SD over 3 random seeds, with the hyperparameters of each model fixed to its best configuration.
| ICU | LLOS | RA | MI | LP | HL | HT | CEL | PC |
|---|---|---|---|---|---|---|---|---|
Comment (12/13):
Does the removal of billing codes extend to DRG codes, and can discharge-coded events that postdate the query time leak into the retrieved context?
For leakage prevention, we exclude all kinds of billing codes that exist in the dataset from appearing in the query segment that serves as a direct input to our model and all baselines, including DRG codes. We have made this statement more clear in Section B.5 Leakage Prevention as follows:
(3) We examine each query if its contents could include any billing codes that might leak labels. Billing codes accounted for are ICD diagnosis codes, ICD procedure codes, DRG codes, as well as MEDS specific codes (i.e. MEDS_DEATH code). ”
Comment (13/13):
What is the code/data URL? The current "link" placeholder prevents verification.
The anonymized code repository is already linked in the manuscript: the word “link” in the contribution section (page 2) is a clickable hyperlink to https://anonymous.4open.science/r/EHR-RAGp-5857. We have now included the URL in full in the text and in the Reproducibility Statement, so that it is also visible in printed copies. We hope these additions address the reviewer’s concerns. We would be grateful if the reviewer would consider revising their score and we are happy to answer any further questions.
Response to Reviewer xmev comments (1/13), (2/13), (3/13), (4/13), (5/13), and (6/13)
We thank Reviewer xmev for the constructive feedback. We are glad that the reviewer appreciates our attention to data-leakage hygiene.
Comment (1/13):
We have addressed each of the soundness concerns raised:
Comment (2/13):
Comment (3/13):
Comment (4/13):
We thank the reviewer for raising this inconsistency.
Updates to Table S4: Best configuration selected for 1-year mortality and In-hospital Mortality after hyperparameter optimization.
Updated Table S13: Effect of the softmax temperature on model performance.
Updated Table S12: Ablation of number of prototypes.
Comment (5/13):
Comment (6/13):
The ICU entropy entries in Table 5 were left over from an earlier 512-prototype run, which is why they exceeded ln 64. We corrected them and they are consistent with all performance results: the ICU configuration uses 64 prototypes, and the entropies are now 4.149 and 4.155 (maximum ln 64 = 4.159). Corrections are shown below for updated rows in Table 5.
Updated rows in Table 5: Ablation of usage regularization on ICU 30-day readmission.