- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes EHR-RAG, a training-free retrieval-augmented framework for clinical prediction from long-horizon structured EHR data. It has three components: event- and time-aware hybrid retrieval (ETHER) combining semantic similarity with a temporal decay score; adaptive iterative retrieval (AIR) refining queries over several rounds; and dual-path evidence retrieval and reasoning (DER) retrieving factual and counterfactual evidence. Evaluation uses four EHRSHOT tasks with GPT-5, Claude-Opus-4.5 and LLaMA-3.1-8B, and the abstract claims an average Macro-F1 improvement of 10.76%.
Critical strengths
The design is sensible and well motivated: preserving event structure and temporal ordering in retrieval is the right instinct for longitudinal EHR data, and the ablation study is reasonably thorough. The paper also attempts variance and self-consistency reporting, and discloses prompts and most hyperparameters, which many submissions in this area do not.
Critical weaknesses
- The headline numbers do not reconcile with the tables: the introduction states a 7.66% gain on Anemia while Table 2 reports +11.68 Macro-F1, and the listed values average about 9.76%, not the 10.76% claimed in the abstract.
- Figure 2 draws the EHR-RAG Acute-MI line near 0.686 although Tables 2-3 report 76.29, which the axis maximum of about 0.70 cannot contain, and two other "over 6 points" gains correspond to +5.83 in Table 4.
- Temporal leakage is not ruled out: for a chunk dated after the prediction time, Eq. (2)'s "recent" term becomes exp(+(tau_c - tau*)/180d) > 1, with no stated cutoff for textual chunks, and pre-tau* filtering is stated only for numeric indicators.
- Because tau* per task is never defined and no split, cohort size, prevalence, or code release is provided, I could not verify that post-prediction evidence is excluded; the case study predicts at 2014-12-05 23:59 while using a GCS reading timestamped 21:15 that day.
- The comparison is not budget-matched in practice: Section 4.2 claims matched inference budgets, but Table 7 shows 96,426.6 tokens for the full method against 15,450.5 and 10,431.9 for baselines, a six- to nine-fold difference.
- Despite the paper's own argument that truncation is the central problem, no full-record long-context baseline is run, and no retrieval-quality metric is reported even though "evidence completeness" is a core selling point.
Questions for authors
- Please define tau* for each task and prediction time, and state explicitly how chunks dated after tau* are excluded for textual evidence.
- Please report test-set size, prevalence, and the exact splits for all four EHRSHOT tasks, and release the code.
- Please reconcile the 10.76% claim with Table 2, and Figure 2's Acute-MI curve with Tables 2-3.
- What are the token counts and inference costs per baseline, and how does the comparison change with a compute-matched or full-context baseline?