acceptodds
Under review as a conference paper at ICLR 2027

EHRAdapt: Adapting Pretrained Language Models to Electronic Health Records with Semantic Priors for Rare Clinical Events

Abstract

Electronic health records (EHRs) encode clinical histories as (time, modality, code) tuples, whereas pretrained language models expect text tokens. Serializing each tuple as text inflates sequence length and repeatedly encodes the same structure. We introduce EHRAdapt, an adapter that maps tuples directly into a frozen language model's embedding space. Modality receives a learned embedding, time gaps enter through learned attention biases, and event codes receive dedicated vectors. Learning these event vectors is the central challenge: clinical vocabularies are long-tailed, leaving rare events with too few observations to estimate reliable representations independently. EHRAdapt therefore represents each event vector as the sum of a semantic prior and an evidence residual. The prior is a frozen embedding of the event's clinical description, from a biomedical language model trained on clinical ontologies, so it carries clinical knowledge. A shared learned projection maps this embedding into the model's input space, allowing the prior to supply clinical meaning even when observations are scarce. The residual is a learned low-rank, event-specific correction that refines this representation as evidence accumulates. We perform continued pretraining on records from about 4 million patients with three pretrained LLMs as backbones: OLMo2 1B, Llama3.2 1B, and OLMo2 7B. As the LLMs remain frozen, only the adapter is trained, which amounts to 0.1–0.6% of all parameters. The full adapter outperforms all ablations in held-out next-event prediction on every backbone. Removing the semantic pathway hurts rare events most, over ten times more than the most frequent ones, whereas removing the residual hurts overall prediction but improves it for the rarest events. On reportable infectious-disease and syndromic downstream classification tasks, EHRAdapt outperforms text-based LLM and count-based baselines, and both pathways improve rare-disease discrimination. The semantic prior and the evidence residual therefore play complementary roles, and these roles become visible only when results are broken down by event frequency rather than averaged.

Primary area:
foundation or frontier models, including LLMs
Keywords:
electronic health records, parameter-efficient adaptation, frozen language models, representation learning, long-tailed distributions, rare events, semantic priors, clinical prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aoz4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EHRAdapt proposes a novel way to map longitudinal electronic health records into the embedding space of a pretrained LLM. They encode time between event with learned attention biases in the LLM. For the embeddings of event they use embeddings from a pretrained biomedical LLM and combine it with an "evidence residual" that is learned via continued autoregressive pretraining on a large scale proprietary dataset of 4M individuals. They evaluate the method within their proprietary cohort on next-event prediction and biosurveillance classification tasks.

Critical strengths

I think the paper is very well presented and the writing is clear. The experiments are well motivated and explained, and the figures are excellent in explaining the method and showcasing the results.

I find the methodological approach, in particular the explicit separation of rare-event information provided by a biomedical language model and a "residual" that can be better learned from longitudinal EHR data very compelling and novel. The ablation in Figure 2 is quite convincing in showing that this decomposition does what it is supposed to do. I think these experiments could lead to many exciting follow up works exploring alternative priors in "hybrid" inductive bias / autoregressive EHR foundation models.

The method performs very well on the tasks and data evaluated by the authors.

Critical weaknesses

The evaluation is limited to one proprietary dataset, so it is very hard to put the results into context. How well do the learned adapters transfer across datasets and healthcare systems? How does the method compare to other state of the art methods on publicly accessible datasets like EHRShot or MIMIC?

Strong baselines are missing in the evaluation, e.g. CLMBR / Mamba-16k (https://arxiv.org/abs/2307.02028, https://arxiv.org/abs/2412.16178) or LLM based approaches (e.g. https://arxiv.org/abs/2502.17403v6). Also only small and not state of the art LLM backbones were used for the evaluation. Taken together with the previous point, this makes it very hard to evaluate how well the method performs.

Questions for authors

How well would the authors expect the method to generalize to other datasets?

How much does the performance depend on the specific biomedical language model? E.g., would a non biomedical but larger or more recent language model perform worse?

Would the method likely perform better with a stronger LLM backbone?

2 comments
AuthorsOpenReview ↗

EHRSHOT transfer and stronger baselines

Thank you for your positive assessment of the presentation and of the semantic-prior/residual decomposition, and for the suggestions on external evaluation. We address each point below.

Q1. Single proprietary dataset; generalization to other systems. We added cross-system transfer to Stanford EHRSHOT on its six new-diagnosis tasks, using the official splits (Sec. 5.3, App. J). The adapter is pretrained only on KPNC; only the logistic-regression probes use EHRSHOT labels. We compare several ways of handling the vocabulary mismatch between KPNC and EHRSHOT. The best performing one uses only the semantic pathway from the Full model and therefore needs no code mapping, reaches macro-AUPRC of 0.167, comparable to 0.160 for Stanford-pretrained CLMBR. We chose EHRSHOT over MIMIC because it provides official splits and labels together with a released foundation-model representation (CLMBR) evaluated under the same probe protocol. Extending the evaluation to MIMIC, and to laboratory-result tasks, which require numeric values absent from our KPNC data, is left for future work.

Q2. Stronger baselines. We added three baselines to the revised manuscript. (1) NEP (Chen et al., 2025) pretrains LoRA adapters on the same backbones and the same KPNC training patients used for EHRAdapt's CPT, with records serialized as text. Like EHRAdapt, this is a self-supervised pretraining stage on unlabeled records; both models are then frozen and evaluated with the same downstream probes. EHRAdapt outperforms it by a large margin on both infectious-disease and syndromic classification, while using fewer GPU-hours (Table 2; App. L). (2) A Hegselmann-style frozen LLM2Vec-Llama-3.1-8B embedding is competitive on low-incidence diseases but clearly below EHRAdapt on syndromic classification (Table 2). (3) CLMBR is compared on EHRSHOT (Q1). A matched-corpus comparison with CLMBR or Mamba models is not possible, because their Stanford pretraining corpus is not available to us.

Q3. Larger or stronger backbones. EHRAdapt accesses the backbone only through input embeddings and attention-score biases, so any decoder can be used without changing the method. The OLMo2 1B-to-7B and Llama 1B-to-8B show no systematic gain over the 1B backbones (Sec. 5.2, App. H). These results suggest that increasing backbone size alone does not necessarily improve downstream performance in this setting.

Q4. Dependence on BioLORD. The framework accepts any fixed per-event vector as the prior; BioLORD is one instantiation, chosen because it embeds each event directly from its description, without mapping local codes to an ontology. Our EHRSHOT experiment illustrates the challenges of transferring event-specific residuals through code mappings: 65.2% of matched diagnosis concepts correspond to multiple KPNC tokens, and only about 18% of input positions receive mapped residuals (App. J). This does not establish that alternative knowledge-graph priors would perform worse. We have not evaluated alternative semantic encoders and acknowledge this limitation.

Thank you for your replies which have largely addressed my concerns and questions, and I do recommend acceptance of the paper.

I did notice that the authors chose CLMBR and LLM2Vec-Llama-3.1-8B for comparison while from both "families" stronger models (Mamba-16k, Qwen3-Emb-8B) exist, and I think it would be worthwhile to include them in the evaluation in the cam-ready. To be clear, my acceptance recommendation is based on the merits and novelty of the method, and does not depend on whether the method outperforms strong baselines across all benchmarks.

Reviewer_53BaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

EHRAdapt explores a simple idea: give every event a text-based embedding based on its description (BioLORD), and learned modifications (residuals) based on its occurrence in the database. Mapped into a "frozen" large language model embedding space improves downstream tasks.

The experiments are well-executed: a complete ablation study across frequency strata reveals interesting patterns. For rare events, the description pathway is critical, while for frequent events learned residuals are informative. One counter intuitive observation is that for rare events, learned residuals may in fact hurt compared to description only pathway.

Critical strengths

The experiment design to investigate role of description pathway and residual pathway is rigorous and sound, and they provide complete results within the text and the appendix.

Critical weaknesses
  • The starting motivation of the paper is that continued pretraining with serialized text is not ideal as it inflates sequence length. However, there is no comparison to a method there. Practically, the paper's overall study is that "if you want to have this specific setup of description-based embedding and learned residual, this is what happens".

  • Some of the conclusions may not generalize beyond the reported specific 64 dimension in Figure 2. CE gains are non-monotone for residual dimensions (Figure 7).

  • From methods point of view, I do not see a clear new developments, and the proposed setup is quite simple. The contributions are mainly related to the experiment design.

Questions for authors
  • The observation that learned residuals hurt the rarest cases is counterintuitive. Does this pattern hold for other other residual dimensions?Can this be an aritifact of your training or test data?

  • Figure 5 reports ~0 accuracy for D1-5 across all setups. How are we really judging full model vs rest in this region?

1 comment
AuthorsOpenReview ↗

Text-serialized baseline, rank robustness, and contribution

Thank you for noting that the experimental design is rigorous and that the frequency-stratified ablations reveal informative patterns. We address each point below.

Q1. No comparison with text-serialized continued pretraining. We added NEP (Chen et al., 2025), which adapts, via continue pre-pretraining, the same three frozen backbones to the same KPNC training patients serialized as text. NEP has a comparable number of trainable parameters and used more GPU-hours than EHRAdapt (App. L, Table 25). On the same backbone, EHRAdapt exceeds NEP (0.380 vs 0.253 AUPRC) for infectious diseases and (0.798 vs 0.732 AUPRC) for syndromic (Table 2). As data and backbone are matched, the gap is not explained by access to EHR data or training budget.

Q2. Does the rare-event residual effect hold beyond rank 64? We extended the analysis to residual ranks 0, 16, 64, 256, and 512 (three seeds each) on the held-out test set. The original Figure 7 is now split into Figure 6 and 7, which extends the semantic-only versus Full comparison of Figure 2 to multiple residual ranks. Increasing rank consistently improves global CE, but for the rarest events (D1–D3) residuals degrade prediction at ranks 16, 64, and 256. The effect is therefore not specific to rank 64 but is a capacity-dependent trade-off that shrinks at larger ranks (Sec. 5.1, App. G).

Q3. Why do residuals hurt the rarest codes, and could this be an artifact? The effect is reproducible across all three backbones and every seed at rank 64 (Fig. 2) and across several residual ranks on OLMo2-1B. It is observed on held-out test patients, with patient-bootstrap CIs excluding zero for pooled D1–D3 at ranks 16–256 (Table 14), and frequency strata are defined on the training split only. These results make an explanation specific to one seed, rank or data split unlikely. Our hypothesis below identifies as the likely cause.

The semantic prior gives every code, including rare ones, a representation informed by biomedical knowledge. Our hypothesis concerns why adding a residual can make rare codes worse than this prior. Because the output head is tied, each code’s residual is updated at every training step, even when the code is absent (Sec. 3.3). For a rare code, almost all of these updates push its score down, and very few come from positions where the code actually occurs. Its residual therefore drifts away from the prior without enough positive evidence to learn a useful correction. The shared projection B is fitted mostly to frequent codes, which can add further noise to rare codes’ residuals. The shrinking penalty at higher ranks suggests that a larger residual space reduces this interference. Our results are consistent with this explanation, but we have not tested the mechanism directly.

Q4. Near-zero top-k accuracy in D1–D5 (original Fig. 5, now Fig. 4). With 78,388 output tokens, rare events almost never reach the top 10, so top-k accuracy saturates. We therefore report cross-entropy, a proper scoring rule that remains informative. As a ranking check, Table 14 reports D1–D3 Recall@1000 with values consistent with the CE result that the residual hurts the sparsest codes.

Q5. Methodological contribution. EHRAdapt is an adapter for frozen LLMs on structured EHRs that represents each event through two complementary pathways: (1) a semantic pathway, which supplies prior biomedical knowledge through a learned projection; and (2) an evidence pathway, which learns a low-rank, event-specific residual from EHR data. We also study how each pathway contributes across event-frequency strata. The semantic pathway matters most for rare events and the evidence pathway for frequent ones, and performance is best with both (Fig. 2; Tables 1 and 16).

This design addresses two problems with existing approaches. First, most methods adapt LLMs to structured EHRs by serializing records as text (Ben Shoham and Rappoport, 2024; Wang et al., 2026; Hegselmann et al., 2026; Chen et al., 2025). Tokenization splits each event into word pieces (~27 tokens per event in our case), lengthening histories and leaving event identity, type and timing implicit. EHRAdapt gives each event a single position, encoding modality and elapsed time with learned embeddings and attention biases.

Second, these approaches do not explicitly address poorly learned rare-event representations, which are common in clinical data. The closest knowledge-based methods each leave a gap. PORTER (Guo et al., 2026) uses fixed BioLORD encodings that cannot adapt to a health system’s data. GRAM (Choi et al., 2017) builds rare-code representations from ontology ancestors, tying it to one ontology hierarchy. EHRAdapt’s evidence pathway adapts each code to system-specific evidence, and its semantic pathway accepts any per-event prior. With BioLORD, events are embedded from their descriptions, so no ontology mapping is needed; knowledge-graph priors can be used where a mapping is available.

Reviewer_gJWjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper tackles the problem of LLMs to natively understand clinical events. The authors tackle this problem trough a medical adapter called EHRAdapt that adapts the clinical events into embeddings that can then be passed to pre-trained LLMs. The events are encoded as the BioLORD embedding of the description, an event type and a modality type embeddings. The model was trained on 4M patients on event (token) prediction, and evaluated on downstream tasks such as disease classification

Critical strengths

The paper introduces a way to incorporate medical event information into pre-trained LLMs without requiring major adaptations in the model architecture/weights, but with an adapter

The way how the information is encoded is simple and elegant since composes a single embedding that summarizes all the patient information in a concise way.

The authors provide multiple evaluations and ablations of the method across diverse backbones and downstream tasks

Critical weaknesses

The adaptation method proposed is only compared agains the frozen LLMs but not against LLMs continuously trained or fine-tuned with the native EHR corpus directly in the input, so is hard to know if the improvement is in the way to represent the data or in the additional model training

The baselines are just frozen LLMs pre-trained for natural domain data so we don't know how the method compares agains a model trained for EHR or adapter for EHR data with other traditional method

Common medical events like lab values and vita-signs are excluded, and numeric values are not represented, which presents an important limitation for real full EHR representation

The decision of time encoding in the model attention can cause problems during the inference for next event prediction because during training the next event time is present in the data and can pass it trough the model, but during inference that time is not there because the next event is generated by the model, but the model doesn't generates time, which limits the method to a single future rollout unless the user synthetically enforces times during inference

Questions for authors
  • Have the authors compared with LLMs adapted with other methods or way to encode the data other than your method and encoding? Maybe comparing with other strategies like continuous pre-training can help us to understand if the method is the one boosting the performance or is just the domain adaptation
  • How you deal with rare events that are not present in the clinical event vocabulary or with the time during multiple inference rollouts?
  • Have the authors done experiments with zero-shot classification? why the decision of downstream performance on the latent space?
1 comment
AuthorsOpenReview ↗

NEP comparison, unseen codes, and zero-shot results

Thank you for recognizing the simplicity of the event encoding and the breadth of our evaluations across backbones and tasks. We address each point below

Q1. Is the gain from the representation or from additional domain training? We added NEP (Chen et al., 2025), which adapts, via continued pre-pretraining, the same three frozen backbones to the same KPNC training patients serialized as text, using the published rank-16 LoRA recipe and its mean pooling. NEP has a comparable number of trainable parameters and used more GPU-hours than EHRAdapt (App. L, Table 25). On the same backbone, EHRAdapt exceeds NEP (0.380 vs 0.253 AUPRC) for infectious diseases and (0.798 vs 0.732 AUPRC) for syndromic (see Table 2). Because the data and backbone are matched, the gap is not explained by access to EHR data or training budget.

We also added a Hegselmann-style frozen LLM2Vec-Llama-3.1-8B embedding. It is competitive on infectious disease (0.371 vs 0.380 AUPRC), but EHRAdapt is clearly stronger on syndromic classification (0.690 vs 0.798 AUPRC). Full results are in Table 2.

Q2. Lab values and vital signs. Laboratory tests and vital-sign measurements are included as events, but their numeric values are not represented, because they were not consistently available in our data. We now state this in Sec. 3.1 and as a limitation in the Conclusions.

Q3. Time encoding and rollouts. EHRAdapt predicts the next event code, not its timestamp, and is not designed for multi-step rollouts, which would require a separate model of elapsed time. Because the hidden state hih_i attends only to positions j ≤ i, the timestamp of the predicted event ci+1c_{i+1} is never an input to its prediction. We clarified this in Sec. 3.4 and limited our claims to one-step prediction and representation learning.

Q4. Unseen codes. In the residual pathway, codes absent from the 2010–2018 KPNC pretraining vocabulary map to a shared <UNK> token. Our audit shows that this affects 0.24–0.40% of input events in 2019 and 4.25–4.57% in 2022, with the highest rates among laboratory events (15.5–16.9%). This may reflect new laboratory codes introduced during the pandemic or changes in coding practice (See full table in Appendix D.3).

Separately, the semantic pathway can represent unseen events directly from their descriptions, via BioLORD, for example. We demonstrate this in the EHRSHOT evaluation, where Stanford events are embedded with the frozen KPNC-trained semantic projection, without code mapping or additional pretraining. This reaches macro-AUPRC of 0.167, comparable to 0.160 for Stanford-pretrained CLMBR (Sec. 5.3).

Q5. Zero-shot classification. We added a zero-shot classification model (Appendix I): Summing next-event probabilities over each disease's codes, however as shown in Table 20 in the appendix, the performance is significantly lower than that achieved by the supervised probes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers and the AC

We thank the reviewers for their careful reading. The three reviews converge on four requests: (1) a comparison with text-serialized adaptation of the same LLMs on the same records, (2) stronger and external baselines, (3) evidence that the frequency-dependent findings are not specific to rank 64 or to our data, and (4) clearer scope for numeric values, timing, and unseen codes. We added the experiments and revisions below; all changes are highlighted in blue in the revised PDF.

AdditionWhere
Next-event prediction (NEP) baseline: same three backbones, same KPNC training patients, published LoRA recipe and poolingSec. 4.4, 5.2; App. L
Hegselmann-style frozen text embedding (LLM2Vec-Llama-3.1-8B)Sec. 4.4, 5.2; App. H
External transfer from KPNC to Stanford EHRSHOT (six diagnosis tasks), compared with Stanford-pretrained CLMBRSec. 5.3; App. J
Held-out residual-rank sweep (ranks 0–512, three seeds, patient-bootstrap CIs, Recall@1000)Sec. 5.1; App. G
Zero-shot disease scoring from the pretrained next-event headSec. 5.2; App. I
Audit of unseen (UNK) codes in the 2019–2022 downstream windowsSec. 4.3; App. D.3
Full adapter on Llama 3.1 8B Base and InstructSec. 5.2; App. H
Clarified temporal causality, numeric values, and narrowed claimsSec. 2, 3.1, 3.4

Terminology. To avoid confusion between "rare diseases" and "rare events", we now call the six infrequent disease classes low-incidence diseases and the corresponding metric group Low-inc. 6 (called "Rare 6" in the original submission); Frequent 7 is unchanged. "Rare events" and "rare codes" keep their meaning: event codes that are infrequent in the training data.

Reorganization and renumbering. To make room for the cross-system analysis, the original Sec. 5.3 (Representation geometry) moved to App. K (Figure 8), with a short summary in Sec. 5.1. As a result, figures and tables were renumbered: original Figure 5 (absolute CPT performance) is now Figure 4; original Figure 7 (residual rank) is now Figures 6–7; original Table 3 (component effects) is now Table 16.

The main new findings are as follows. First, EHRAdapt outperforms NEP on every downstream view on the same backbone and data, although NEP used more GPU-hours. Second, EHRAdapt has the strongest syndromic performance of all evaluated methods, whereas off-the-shelf 8B text embeddings are competitive on low-incidence-disease labels. Third, the semantic pathway of an adapter pretrained only on KPNC transfers to Stanford without code mapping, with macro-AUPRC comparable to Stanford-pretrained CLMBR. Fourth, the residual-rank sweep shows that the frequency-dependent behavior is not specific to rank 64: residuals degrade the rarest codes at ranks 16, 64, and 256, and the penalty shrinks with capacity until it is no longer significant at rank 512.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.