acceptodds
Under review as a conference paper at ICLR 2027

EHR-RAG: Bridging Long-Horizon Structured Electronic Health Records and Large Language Models via Enhanced Retrieval-Augmented Generation

Abstract

Electronic Health Records (EHRs) provide rich longitudinal clinical evidence that is central to medical decision-making, motivating the use of retrieval-augmented generation (RAG) to ground large language model (LLM) predictions. However, long-horizon EHRs often exceed LLM context limits, and existing approaches commonly rely on truncation or vanilla retrieval strategies that discard clinically relevant events and temporal dependencies. To address these challenges, we propose EHR-RAG, a retrieval-augmented framework designed for accurate interpretation of long-horizon structured EHR data. EHR-RAG introduces three components tailored to longitudinal clinical prediction tasks: Event- and Time-Aware Hybrid EHR Retrieval to preserve clinical structure and temporal dynamics, Adaptive Iterative Retrieval to progressively refine queries in order to expand broad evidence coverage, and Dual-Path Evidence Retrieval and Reasoning to jointly retrieves and reasons over both factual and counterfactual evidence to mitigate bias toward evidence supporting only one outcome. Experiments across four long-horizon EHR prediction tasks show that EHR-RAG consistently outperforms the strongest LLM-based baselines, achieving an average Macro-F1 improvement of 10.76%. Overall, our work highlights the potential of retrieval-augmented LLMs to advance clinical prediction on structured EHR data in practice.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Electronic Health Record, Clinical Prediction, Retrieval-Augmented Generation, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8MchOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EHR-RAG, a training-free retrieval-augmented framework for clinical prediction from long-horizon structured EHR data. It has three components: event- and time-aware hybrid retrieval (ETHER) combining semantic similarity with a temporal decay score; adaptive iterative retrieval (AIR) refining queries over several rounds; and dual-path evidence retrieval and reasoning (DER) retrieving factual and counterfactual evidence. Evaluation uses four EHRSHOT tasks with GPT-5, Claude-Opus-4.5 and LLaMA-3.1-8B, and the abstract claims an average Macro-F1 improvement of 10.76%.

Critical strengths

The design is sensible and well motivated: preserving event structure and temporal ordering in retrieval is the right instinct for longitudinal EHR data, and the ablation study is reasonably thorough. The paper also attempts variance and self-consistency reporting, and discloses prompts and most hyperparameters, which many submissions in this area do not.

Critical weaknesses
  • The headline numbers do not reconcile with the tables: the introduction states a 7.66% gain on Anemia while Table 2 reports +11.68 Macro-F1, and the listed values average about 9.76%, not the 10.76% claimed in the abstract.
  • Figure 2 draws the EHR-RAG Acute-MI line near 0.686 although Tables 2-3 report 76.29, which the axis maximum of about 0.70 cannot contain, and two other "over 6 points" gains correspond to +5.83 in Table 4.
  • Temporal leakage is not ruled out: for a chunk dated after the prediction time, Eq. (2)'s "recent" term becomes exp(+(tau_c - tau*)/180d) > 1, with no stated cutoff for textual chunks, and pre-tau* filtering is stated only for numeric indicators.
  • Because tau* per task is never defined and no split, cohort size, prevalence, or code release is provided, I could not verify that post-prediction evidence is excluded; the case study predicts at 2014-12-05 23:59 while using a GCS reading timestamped 21:15 that day.
  • The comparison is not budget-matched in practice: Section 4.2 claims matched inference budgets, but Table 7 shows 96,426.6 tokens for the full method against 15,450.5 and 10,431.9 for baselines, a six- to nine-fold difference.
  • Despite the paper's own argument that truncation is the central problem, no full-record long-context baseline is run, and no retrieval-quality metric is reported even though "evidence completeness" is a core selling point.
Questions for authors
  1. Please define tau* for each task and prediction time, and state explicitly how chunks dated after tau* are excluded for textual evidence.
  2. Please report test-set size, prevalence, and the exact splits for all four EHRSHOT tasks, and release the code.
  3. Please reconcile the 10.76% claim with Table 2, and Figure 2's Acute-MI curve with Tables 2-3.
  4. What are the token counts and inference costs per baseline, and how does the comparison change with a compute-matched or full-context baseline?
Reviewer_GZ6qOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EHR-RAG, a framework for long-horizon EHR prediction that integrates event- and time-aware retrieval, iterative query refinement, and dual-path evidence reasoning. Experiments on four EHRSHOT tasks show consistent improvements over LLM-based baselines.

Critical strengths

Interesting and practically relevant problem. The paper explores an interesting direction by applying retrieval-augmented reasoning to long-horizon structured EHR data. This is a meaningful problem, as clinically relevant information is often scattered across lengthy patient histories.

Critical weaknesses
  1. Insufficient comparison with competitive EHR-specific baselines.

The main experiments primarily compare EHR-RAG against generic RAG methods that are not designed for structured longitudinal EHR prediction. Although additional baselines are included in the appendix, comparisons with stronger EHR-specific approaches remain limited. This makes it difficult to assess the actual advantage of the proposed framework over existing methods.

  1. Unfair inference budget and unclear source of improvements.

EHR-RAG introduces multiple retrieval and reasoning stages, resulting in substantially higher token consumption and inference latency than the baselines. Without rigorous budget-matched comparisons, it is unclear whether the improvements stem from the proposed designs or simply from increased inference-time computation.

Questions for authors
  1. Inconsistent reported results. The Introduction reports a 7.66-point Macro-F1 improvement on Anemia, while Table 2 shows an improvement of 11.68 points. Could the authors clarify this discrepancy and verify the reported results?

  2. EHR-specific baselines. Why are the main comparisons primarily limited to generic RAG methods? Could the authors provide comparisons with stronger EHR-specific prediction or reasoning methods to better demonstrate the effectiveness of EHR-RAG?

  3. Inference budget fairness. EHR-RAG consumes substantially more tokens and inference time than the baselines. Could the authors provide budget-matched comparisons to clarify whether the improvements come from the proposed components rather than additional inference-time computation?

  4. Long-horizon memory management.Although Appendix J includes several memory-based baselines, why are more explicit long-horizon memory management, compression, or agent-harness approaches not considered? How would EHR-RAG compare with methods that maintain and selectively access compressed patient histories under similar computational budgets?

Reviewer_9wpjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on how to use LLM+rag to predict clinical decision via long-horizon structured EHR data. EHR-RAG combines event and time-aware hybrid EHR retrieval, adaptive iterative retrieval and dual-path evidence retrieval and reasoning. Event and time-aware hybrid EHR retrieval aggregates numerical event history as a indicator-level trajectory, retrieve events with a U-shape early and recent temporal prior. Adaptive iterative retrieval uses iterative query to refine the evidence coverage and sufficiency. Dual-path evidence retrieval and reasoning provides two prediction hypothesis evidence retrieval and reasoning and then fuse the evidences.

The main GPT-5 experiments covers 4 EHRSHOT tasks: long length of stay, 30-day readmission, new acute myocardial infarction, and four-class anemia prediction. Tables 1 and 2 report macro-F1 gains of 3.63%, 11.28%, 16.46%, and 11.68% over the strongest listed LLM baseline. Additional experiments examine other LLM backbones, other data modeling methods like count-based and CLMBR-based logistic regression, and efficiency (cost and latency)

Critical strengths

S1. EHR-RAG addresses concrete properties of long-horizon structured EHRs. Events in a longitudinal record need not receive equal weight. EHR-RAG provides an inspectable way to organize and retrieve numerical event trajectories, recurring events, and evidence from early and recent history. It retrieves this information without fine-tuning the LLM. This is useful when labeled examples for a clinical task are limited or when traditional similarity-based retrieval may overlook relevant evidence.

S2. The evaluation extends beyond a single task or result table. Tables 1–2 report per-class F1 and macro-F1 scores on four different prediction tasks, including substantial reported minority-class improvements. The appendices include an additional dataset, ML comparisons, broader RAG/memory controls, cost measurements, and repeated runs. These are meaningful strengths, even though the problems below currently limit the conclusions that can be drawn from them.

Critical weaknesses

W1. Tables 2 and 3 contain metric reporting issues.

W1.1. In Table 2, the RAG row for Anemia reports per-class F1 scores of 70.43, 18.85, 12.90, and 0.00. Their mean is 25.545%, whereas the reported macro-F1 is 20.44%.

W1.2. Tables 2 and 3 show a conditional cross-table inconsistency. Table 2 reports an Acute MI accuracy of 92.95% and a positive-class F1 of 56.41% for EHR-RAG. Table 3 reports an Acute MI accuracy of 66.67% and a macro-F1 of 62.50% for w/o DER. If each row is computed from a single pooled binary confusion matrix on the same testset, the former requires a positive prevalence of at most approximately 11.6209%, while the latter requires at least approximately 16.6453%, allowing for rounding of the displayed values. Those conditions cannot hold simultaneously.

W2. The experimental evidence does not yet establish the mechanisms and comparative value of the proposed design.

W2.1 — The claimed budget matching is not established by the reported configurations. Appendix D states, “we fix the maximum context length to 128,000 tokens,” but how this limit is enforced and what is held constant by “matched inference budgets” remain unclear. Direct Generation retains 1,000 events, while vanilla RAG retrieves ten chunks. In the supplied code, EHR-RAG begins with ten chunks per hypothesis, adds recent chunks and numerical evidence, and can refine both paths. These configurations allow different amounts of retrieval and inference. Table 7 reports 96,426.6 average input tokens for EHR-RAG versus 15,450.5 for vanilla RAG and 10,431.9 for ReAct using GPT-5-nano on the Long Length of Stay task. These measurements show different cumulative usage in that setting; they do not establish whether the main GPT-5 comparisons use comparable resources.

In the supplied GPT code, max_tokens is set to 128,000 and passed as a generation-length limit, which does not by itself enforce an input context limit. Please clarify how the stated context limit is enforced and whether a separate cumulative inference budget is applied. I suggest reporting cumulative input/output tokens, the number of LLM calls, and the amount of unique EHR evidence for all compared methods in the main GPT-5 experiments.

I also suggest reporting the total cost per prediction for GPT-5 to complement the existing GPT-5-nano analysis.

W2.2 — ETHER's U-shaped temporal prior is not isolated in the ablation study. Table 3 removes the whole ETHER component, so it does not identify whether the gain comes from numerical histories, temporal reranking, or their combination. The earliest recorded clinical event used by the code is not necessarily the onset of a task-relevant condition. The motivation for the U-shaped prior (page 5) and the claimed hyperparameter insensitivity (page 17) therefore need direct support. I suggest separating numerical trajectory organization from temporal reranking, then comparing semantic-only, recency-only, and U-shaped scoring while holding the other evidence-selection settings fixed. A limited sensitivity analysis would test the claim about the chosen settings.

W2.3 — The benefit of reasoning about opposing hypotheses is not separated from the benefit of potentially retrieving more evidence and making extra calls.

Removing DER changes evidence coverage as well as the reasoning procedure. Appendix I's overlap analysis identifies differences between retrieved sets, but does not establish that those differences are useful or that reasoning about opposing hypotheses causes the gain. Including DER adds calls for a second reasoning path and can increase the amount of retrieved evidence. I suggest holding the evidence set fixed when comparing neutral reasoning, two paths supporting the same hypothesis, and two paths supporting opposing hypotheses. Separately, retrieval through one path and two paths could be compared using the same reasoning procedure.

W2.4 — The use of two path hypotheses is not sufficiently justified for four-class Anemia prediction.

For Anemia, the final fusion can output any of the four labels, but the two reasoning paths argue only for labels 3 and 0. The paths do not explicitly construct hypotheses for the intermediate classes, labels 1 and 2, and the paper does not establish whether this design adequately supports discrimination between them. Meanwhile, the generic fusion instruction in the code defaults to label 0 when the evidence on both sides is weak or balanced. The Anemia-specific instruction prefers 1 or 2 when severity is uncertain. These instructions can prescribe conflicting defaults when their conditions overlap, and their priority in the fusion prompt is unspecified.

W2.5 — The selection of four EHRSHOT tasks is not sufficiently explained.

The four tasks cover three task families, but the selection criterion is unspecified. This limits the assessment of cross-task generality and potential selection bias. I suggest stating the selection and exclusion criteria and broadening coverage where feasible. Evaluating every model and ablation on every task is not requested as a prerequisite.

Questions for authors

Q1. Anemia metric (W1.1). What exact label set and invalid-output policy produced the Table 2 RAG row? Please provide the actual/predicted label counts and confusion matrix, and identify how the four displayed F1 scores relate to macro-F1 20.44. If correction is needed, please update the affected tables and gains.

Q2. Acute-MI aggregation and shared instances (W1.2). Are Tables 2–3 computed from pooled predictions or means of per-run/group metrics? Please provide the instance counts, positive/negative counts, per-run confusion matrices, aggregation weights, and failed/missing-output handling for the full and ablated methods. An anonymized manifest or hashes should establish matching patient–prediction-time instances. A reproducible mean-of-runs explanation using identical instances would resolve the conditional inconsistency.

Q3. Resource accounting and mechanism controls (W2.1–W2.3). Which configurations produced the main tables and ablations, and what exactly was held constant by “matched inference budgets”? Please account for initial retrieval, both paths, query refinement, numerical reranking, and fusion. Do existing results or targeted controls distinguish the benefits of the U-shaped prior and reasoning about opposing hypotheses from the benefits of additional evidence and calls?

Q4. Multiclass DER and decision defaults (W2.4). What motivates supporting only labels 0 and 3 in the Anemia reasoning paths, and how are the overlapping generic and task-specific decision instructions resolved? Per-class confusion matrices and a neutral multiclass control on the same evidence would help understand whether the observed benefit comes from the competing hypotheses or the task-specific decision preferences.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.