- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
TabEHR is a method that tries to bring the recent successes of tabular foundation models such as TabPFN to longitudinal EHR data. The core idea of the authors is to use a pretrained TFM (EXAONE), and via a multi stage process, learn semantically meaningful concept embeddings that can then be used to map longitudinal patient EHR data into the input space of tabular foundation models. This way only the semantic mapping has to be learned and the TFM can stay fixed. The authors pretrain this on one specific dataset (MIMIC-IV) and evaluate performance both within MIMIC-IV and also to the external EHRSHOT cohort that was not used for pretraining. The semantic embeddings are derived via MLM and contrastive learning on LLM generated concept descriptions of OMOP terms.
Critical strengths
The specific approach the authors proposed to derive semantic embeddings of EHR / OMOP terms is quite interesting and to my knowledge novel. It is still an open research question how EHR can be preprocessed for use in foundation models on EHR data in a way that generalizes across different healthcare systems and institution.
Critical weaknesses
Most current work on foundation models for EHR data explicitely includes the temporal, longitudinal information. E.g., autoregressive models such as CLMBR or DELPHI model the sequence of EHR. Likewise, approaches that encode EHR as text and feed them into LLMs or MLM embedding models usually represent temporal information in the text. While TabEHR does use temporal binning similar to counts based methods, it is questionable to me if this can really preserve the meaningful temporal information in the data, and as far as I can tell, no ablations explicitly investigate this further. I do think that SCM like pretrained for EHR foundation models is an exciting direction, but this would likely require the TFM pretraining to include temporal dependencies instead of using a fixed TFM.
The overall method has many components, but it is not clear to me what actually drives the performance. Critical ablations are missing. Likewise, other EHR foundation models such as CLMBR / Mamba-16k (https://arxiv.org/abs/2307.02028, https://arxiv.org/abs/2412.16178) or LLM based approaches (e.g. https://arxiv.org/abs/2502.17403v6) are not being evaluated. E.g. both Mamba-16k and Qwen3-Embedding-8B seem to perform at least as good as TabEHR in the few shot experiment in table 2.
Questions for authors
The method was partially pretrained on MIMIC-IV. To what extent is the performance of the method driven by this specific cohort? How well would the authors expect the method generalizes, e.g. to settings outside of the US healthcare system?
Why was the few shot prediction in Table 2 only performed on k=16?
Have other pretrained TFMs been tested, or is the approach specific to EXAONE Tabular?
How circular is the concept embedding evaluation? Is this testing the embedding model on relationships that were explicitly provided during pretraining?
6 comments
Response to Reviewer wv6w (2/5)
W2-a. Component attribution
W2: “The overall method has many components, but it is not clear to me what actually drives the performance. Critical ablations are missing.”
We examined the components one at a time. The table lists what changes on EHRSHOT when each component is replaced, with the frozen TFM kept fixed.
| Comparison | What changes | EHRSHOT AUROC / AUPRC |
|---|---|---|
| MEDS-Tab + TFM → BiomedBERT + TFM | statistical features → patient features learned from events, with concept-name embeddings | .746/.432 → .779/.454 |
| BiomedBERT + TFM → TabEHR | concept-name embeddings → TabEHR concept embeddings | .779/.454 → .804/.510 |
| TabEHR → random vectors as anchors | clinical meaning of the reference points | .811/.525 → .756/.391 |
| TabEHR → without the similarity term | concept–anchor similarity in the attention score | .811/.525 → .679/.281 |
| TabEHR → without the learned term | learned term in the attention score | .811/.525 → .814/.521 |
| TabEHR → recency selection | coverage-first token selection | .811/.525 → .805/.496 |
| TabEHR → without recency encoding | elapsed time of each token | .811/.525 → .788/.495 |
Nine-task macro scores. The first two comparisons are three-seed means from Table 1; the others use one training seed (100). For reference, the between-seed SD of TabEHR on EHRSHOT is .015/.027.
Three results follow. First, learning patient features from events improves transfer over statistical features even with public concept-name embeddings (+.033/+.022), and the TabEHR concept embeddings add a further +.025/+.056. Second, within the aggregation, the gain depends on clinically meaningful anchors and on the concept–anchor similarity term: removing either lowers external AUROC by .055 and .132. Third, the learned term in the attention score is not required for this gain. Among the event inputs, removing recency encoding lowers external AUROC/AUPRC by .023/.030 (W1), and replacing coverage-first with recency selection lowers them by .006/.029. The dependence on the TFM backbone is addressed in Q3.
Manuscript revision. Section 4.3 now reports the aggregation controls, and Section 3.4 points to them and to the sensitivity to the number of anchors in Appendix I.4.
Response to Reviewer wv6w (3/5)
W2-b. Additional comparators
W2: “Likewise, other EHR foundation models such as CLMBR / Mamba-16k (https://arxiv.org/abs/2307.02028, https://arxiv.org/abs/2412.16178) or LLM based approaches (e.g. https://arxiv.org/abs/2502.17403v6) are not being evaluated. E.g. both Mamba-16k and Qwen3-Embedding-8B seem to perform at least as good as TabEHR in the few shot experiment in table 2.”
We added the comparators you list: Qwen3-Embedding-8B and CLMBR-T-base evaluated with the same frozen TFM, and the published Mamba-16k results, at the five support sizes of the general response. Table A compares models without target-institution pretraining on the fifteen tasks; Table B compares TabEHR with models pretrained on EHRs of the target institution.
Table A. Few-shot transfer to an external institution with a shared TFM (Overall, 15 tasks).
| Patient representation | k=8 | k=16 | k=32 | k=64 | k=128 |
|---|---|---|---|---|---|
| Count | .589 | .616 | .647 | .671 | .693 |
| MedRep + BEHRT (FT) | .597 ± .004 | .622 ± .002 | .639 ± .003 | .651 ± .001 | .665 ± .002 |
| MedTok + BEHRT (FT) | .596 ± .010 | .628 ± .008 | .647 ± .007 | .668 ± .009 | .686 ± .004 |
| Qwen3-Embedding-8B | .643 | .667 | .683 | .701 | .721 |
| TabEHR | .589 ± .013 | .643 ± .022 | .698 ± .025 | .733 ± .018 | .750 ± .011 |
Macro AUROC over five support replicates with the same frozen TFM. Fine-tuned (FT) models and TabEHR report mean ± sample SD across three source-training seeds. Qwen3-Embedding-8B is used without additional EHR training. CXR averages 14 findings as one task. Bold: highest mean within each k.
As you observed, Qwen3-Embedding-8B is higher at small support sizes (k=8 and k=16). From k=32 TabEHR is higher over all tasks, and at k=128 it reaches .750/.324 in AUROC/AUPRC against .721/.285. From k=64, TabEHR is higher than Qwen3-Embedding-8B in every task group on both metrics.
Table B. Comparison with models pretrained on target-institution EHRs.
| Model | Target-institution EHR pretraining | Predictor | k=8 | k=16 | k=32 | k=64 | k=128 |
|---|---|---|---|---|---|---|---|
| CLMBR-T-base | O | TFM | .617 | .643 | .670 | .685 | .717 |
| Mamba-16k (reported) | O | LR | ≈.654 | ≈.681 | ≈.714 | ≈.738 | ≈.766 |
| TabEHR | X | TFM | .591 ± .014 | .647 ± .023 | .704 ± .027 | .740 ± .019 | .773 ± .011 |
All rows are re-aggregated to the task sets of the Mamba report: 14 tasks without CXR at k=8–64, and 13 tasks, also without Celiac, at k=128. Mamba values are reconstructed from published task-group means (Context Clues, Appendix G and Tables 11–13). Mamba + LR uses k validation labels per class in addition to k training labels; the TFM rows use k labels per class. Bold: highest value within each k.
CLMBR-T-base and Mamba-16k are pretrained on EHRs of the target institution, and TabEHR is not. With the same frozen TFM, TabEHR is similar to CLMBR-T-base at k=16 and higher from k=32. At k=128 on this task set, TabEHR has the highest AUROC and AUPRC among the directly evaluated representations (Appendix J.5): .773/.359, against .736/.316 for Qwen3-Embedding-8B and .717/.283 for CLMBR-T-base. The published Mamba-16k values are higher at k=8–32; at k=64 and k=128, TabEHR reaches the same level (.740 and .773 against ≈.738 and ≈.766) with half as many target labels per class.
Manuscript revision. Appendix J.5 now reports the task-aligned comparison at k=16 and k=128 in AUROC and AUPRC, and marks which models are pretrained on target-institution EHRs and which values are taken from the literature.
Response to Reviewer wv6w (1/5)
We thank Reviewer wv6w for the thoughtful comments. General responses C3, C1, and C2 give the full results for Q1–Q2 (few-shot transfer), Q3 (other TFM backbones), and the role of the anchors in prediction (Q4); we answer each point below.
W1. Temporal information and the frozen TFM
TabEHR encodes when each observation occurred, and this timing reaches the frozen TFM through the cells built from the event tokens. We agree that including temporal dependencies in TFM pretraining is a promising direction. Our claim is narrower: temporal information can be used for prediction with a frozen TFM.
What is preserved. Records of the same concept are grouped into elapsed-time bins, which keep daily resolution for recent observations and use wider intervals further in the past. Each concept–bin token stores its record count, the most recent non-missing value, the minimum and maximum non-missing values, and the timestamp of its most recent record, and recency encoding adds the elapsed time between that record and the prediction time. Each token thus carries what was observed and when.
How it reaches the predictor. The frozen TFM receives one row per prediction instance, not a sequence of rows. The event encoder and anchor aggregator build the cells of that row from the tokens, and because they are trained through the TFM's prediction loss, they can learn to represent timing in a form useful to that predictor.
Evidence. Removing recency encoding, with temporal binning and token selection kept, lowers macro AUROC/AUPRC by .015/.030 on MIMIC-IV and by .023/.030 on EHRSHOT:
Nine-task macro averages with one training seed (100); checkpoints are selected by MIMIC-IV validation macro AUROC. Bold: best in each column.
Both decreases exceed the between-seed SD of TabEHR (.003/.004 on MIMIC-IV and .015/.027 on EHRSHOT). In Table 1, TabEHR is also higher than every evaluated EHR sequence model, which processes the ordered event sequence directly, in both cohorts: .899/.558 against at most .876/.487 on MIMIC-IV, and .804/.510 against at most .762/.423 on EHRSHOT.
Manuscript revision. Section 3.3 now states how timing is encoded in each token and how it reaches the frozen TFM.