- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EHRAdapt proposes a novel way to map longitudinal electronic health records into the embedding space of a pretrained LLM. They encode time between event with learned attention biases in the LLM. For the embeddings of event they use embeddings from a pretrained biomedical LLM and combine it with an "evidence residual" that is learned via continued autoregressive pretraining on a large scale proprietary dataset of 4M individuals. They evaluate the method within their proprietary cohort on next-event prediction and biosurveillance classification tasks.
Critical strengths
I think the paper is very well presented and the writing is clear. The experiments are well motivated and explained, and the figures are excellent in explaining the method and showcasing the results.
I find the methodological approach, in particular the explicit separation of rare-event information provided by a biomedical language model and a "residual" that can be better learned from longitudinal EHR data very compelling and novel. The ablation in Figure 2 is quite convincing in showing that this decomposition does what it is supposed to do. I think these experiments could lead to many exciting follow up works exploring alternative priors in "hybrid" inductive bias / autoregressive EHR foundation models.
The method performs very well on the tasks and data evaluated by the authors.
Critical weaknesses
The evaluation is limited to one proprietary dataset, so it is very hard to put the results into context. How well do the learned adapters transfer across datasets and healthcare systems? How does the method compare to other state of the art methods on publicly accessible datasets like EHRShot or MIMIC?
Strong baselines are missing in the evaluation, e.g. CLMBR / Mamba-16k (https://arxiv.org/abs/2307.02028, https://arxiv.org/abs/2412.16178) or LLM based approaches (e.g. https://arxiv.org/abs/2502.17403v6). Also only small and not state of the art LLM backbones were used for the evaluation. Taken together with the previous point, this makes it very hard to evaluate how well the method performs.
Questions for authors
How well would the authors expect the method to generalize to other datasets?
How much does the performance depend on the specific biomedical language model? E.g., would a non biomedical but larger or more recent language model perform worse?
Would the method likely perform better with a stronger LLM backbone?
2 comments
Thank you for your replies which have largely addressed my concerns and questions, and I do recommend acceptance of the paper.
I did notice that the authors chose CLMBR and LLM2Vec-Llama-3.1-8B for comparison while from both "families" stronger models (Mamba-16k, Qwen3-Emb-8B) exist, and I think it would be worthwhile to include them in the evaluation in the cam-ready. To be clear, my acceptance recommendation is based on the merits and novelty of the method, and does not depend on whether the method outperforms strong baselines across all benchmarks.
EHRSHOT transfer and stronger baselines
Thank you for your positive assessment of the presentation and of the semantic-prior/residual decomposition, and for the suggestions on external evaluation. We address each point below.
Q1. Single proprietary dataset; generalization to other systems. We added cross-system transfer to Stanford EHRSHOT on its six new-diagnosis tasks, using the official splits (Sec. 5.3, App. J). The adapter is pretrained only on KPNC; only the logistic-regression probes use EHRSHOT labels. We compare several ways of handling the vocabulary mismatch between KPNC and EHRSHOT. The best performing one uses only the semantic pathway from the Full model and therefore needs no code mapping, reaches macro-AUPRC of 0.167, comparable to 0.160 for Stanford-pretrained CLMBR. We chose EHRSHOT over MIMIC because it provides official splits and labels together with a released foundation-model representation (CLMBR) evaluated under the same probe protocol. Extending the evaluation to MIMIC, and to laboratory-result tasks, which require numeric values absent from our KPNC data, is left for future work.
Q2. Stronger baselines. We added three baselines to the revised manuscript. (1) NEP (Chen et al., 2025) pretrains LoRA adapters on the same backbones and the same KPNC training patients used for EHRAdapt's CPT, with records serialized as text. Like EHRAdapt, this is a self-supervised pretraining stage on unlabeled records; both models are then frozen and evaluated with the same downstream probes. EHRAdapt outperforms it by a large margin on both infectious-disease and syndromic classification, while using fewer GPU-hours (Table 2; App. L). (2) A Hegselmann-style frozen LLM2Vec-Llama-3.1-8B embedding is competitive on low-incidence diseases but clearly below EHRAdapt on syndromic classification (Table 2). (3) CLMBR is compared on EHRSHOT (Q1). A matched-corpus comparison with CLMBR or Mamba models is not possible, because their Stanford pretraining corpus is not available to us.
Q3. Larger or stronger backbones. EHRAdapt accesses the backbone only through input embeddings and attention-score biases, so any decoder can be used without changing the method. The OLMo2 1B-to-7B and Llama 1B-to-8B show no systematic gain over the 1B backbones (Sec. 5.2, App. H). These results suggest that increasing backbone size alone does not necessarily improve downstream performance in this setting.
Q4. Dependence on BioLORD. The framework accepts any fixed per-event vector as the prior; BioLORD is one instantiation, chosen because it embeds each event directly from its description, without mapping local codes to an ontology. Our EHRSHOT experiment illustrates the challenges of transferring event-specific residuals through code mappings: 65.2% of matched diagnosis concepts correspond to multiple KPNC tokens, and only about 18% of input positions receive mapped residuals (App. J). This does not establish that alternative knowledge-graph priors would perform worse. We have not evaluated alternative semantic encoders and acknowledge this limitation.