acceptodds
Under review as a conference paper at ICLR 2027

TABEHR: BRIDGING LONGITUDINAL EHRS AND TABULAR FOUNDATION MODELS

Abstract

Longitudinal electronic health records (EHRs) are heterogeneous and irregular: patients differ in which clinical concepts are recorded, when they are observed, and how often they occur, and these recording patterns also vary across institutions. This makes it difficult to reuse clinical prediction models across sites. Tabular foundation models (TFMs) offer reusable in-context prediction from labeled support patients without parameter updates, but require corresponding feature columns that irregular EHR histories do not provide. We introduce TabEHR, which learns this missing interface by mapping irregular histories into shared semantic columns for a frozen TFM. TabEHR constructs clinically enriched concept embeddings from ontology-informed descriptions synthesized through four complementary clinical perspectives and uses their geometry to define fixed semantic anchors that aggregate patient events into column-specific embeddings. Keeping the anchors and TFM fixed, we train TabEHR on MIMIC-IV to learn how patient events populate these shared columns. TabEHR achieves the highest macro-averaged AUROC and AUPRC across nine tasks among the evaluated methods, both on held-out MIMIC-IV patients and on the external EHRSHOT cohort, without using EHRSHOT outcome labels.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Electronic Health Records (EHR), Tabular Foundation Models, In-Context Learning, Longitudinal Clinical Data, OMOP Common Data Model, Clinical Prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wv6wOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

TabEHR is a method that tries to bring the recent successes of tabular foundation models such as TabPFN to longitudinal EHR data. The core idea of the authors is to use a pretrained TFM (EXAONE), and via a multi stage process, learn semantically meaningful concept embeddings that can then be used to map longitudinal patient EHR data into the input space of tabular foundation models. This way only the semantic mapping has to be learned and the TFM can stay fixed. The authors pretrain this on one specific dataset (MIMIC-IV) and evaluate performance both within MIMIC-IV and also to the external EHRSHOT cohort that was not used for pretraining. The semantic embeddings are derived via MLM and contrastive learning on LLM generated concept descriptions of OMOP terms.

Critical strengths

The specific approach the authors proposed to derive semantic embeddings of EHR / OMOP terms is quite interesting and to my knowledge novel. It is still an open research question how EHR can be preprocessed for use in foundation models on EHR data in a way that generalizes across different healthcare systems and institution.

Critical weaknesses

Most current work on foundation models for EHR data explicitely includes the temporal, longitudinal information. E.g., autoregressive models such as CLMBR or DELPHI model the sequence of EHR. Likewise, approaches that encode EHR as text and feed them into LLMs or MLM embedding models usually represent temporal information in the text. While TabEHR does use temporal binning similar to counts based methods, it is questionable to me if this can really preserve the meaningful temporal information in the data, and as far as I can tell, no ablations explicitly investigate this further. I do think that SCM like pretrained for EHR foundation models is an exciting direction, but this would likely require the TFM pretraining to include temporal dependencies instead of using a fixed TFM.

The overall method has many components, but it is not clear to me what actually drives the performance. Critical ablations are missing. Likewise, other EHR foundation models such as CLMBR / Mamba-16k (https://arxiv.org/abs/2307.02028, https://arxiv.org/abs/2412.16178) or LLM based approaches (e.g. https://arxiv.org/abs/2502.17403v6) are not being evaluated. E.g. both Mamba-16k and Qwen3-Embedding-8B seem to perform at least as good as TabEHR in the few shot experiment in table 2.

Questions for authors

The method was partially pretrained on MIMIC-IV. To what extent is the performance of the method driven by this specific cohort? How well would the authors expect the method generalizes, e.g. to settings outside of the US healthcare system?

Why was the few shot prediction in Table 2 only performed on k=16?

Have other pretrained TFMs been tested, or is the approach specific to EXAONE Tabular?

How circular is the concept embedding evaluation? Is this testing the embedding model on relationships that were explicitly provided during pretraining?

6 comments
AuthorsOpenReview ↗

Response to Reviewer wv6w (1/5)

We thank Reviewer wv6w for the thoughtful comments. General responses C3, C1, and C2 give the full results for Q1–Q2 (few-shot transfer), Q3 (other TFM backbones), and the role of the anchors in prediction (Q4); we answer each point below.

W1. Temporal information and the frozen TFM

W1: “…While TabEHR does use temporal binning similar to counts based methods, it is questionable to me if this can really preserve the meaningful temporal information in the data … this would likely require the TFM pretraining to include temporal dependencies instead of using a fixed TFM.”

TabEHR encodes when each observation occurred, and this timing reaches the frozen TFM through the cells built from the event tokens. We agree that including temporal dependencies in TFM pretraining is a promising direction. Our claim is narrower: temporal information can be used for prediction with a frozen TFM.

What is preserved. Records of the same concept are grouped into elapsed-time bins, which keep daily resolution for recent observations and use wider intervals further in the past. Each concept–bin token stores its record count, the most recent non-missing value, the minimum and maximum non-missing values, and the timestamp of its most recent record, and recency encoding adds the elapsed time between that record and the prediction time. Each token thus carries what was observed and when.

How it reaches the predictor. The frozen TFM receives one row per prediction instance, not a sequence of rows. The event encoder and anchor aggregator build the cells of that row from the tokens, and because they are trained through the TFM's prediction loss, they can learn to represent timing in a form useful to that predictor.

Evidence. Removing recency encoding, with temporal binning and token selection kept, lowers macro AUROC/AUPRC by .015/.030 on MIMIC-IV and by .023/.030 on EHRSHOT:

Event inputMIMIC-IV AUROC / AUPRCEHRSHOT AUROC / AUPRC
TabEHR.900 / .562.811 / .525
Recency encoding removed.885 / .532.788 / .495

Nine-task macro averages with one training seed (100); checkpoints are selected by MIMIC-IV validation macro AUROC. Bold: best in each column.

Both decreases exceed the between-seed SD of TabEHR (.003/.004 on MIMIC-IV and .015/.027 on EHRSHOT). In Table 1, TabEHR is also higher than every evaluated EHR sequence model, which processes the ordered event sequence directly, in both cohorts: .899/.558 against at most .876/.487 on MIMIC-IV, and .804/.510 against at most .762/.423 on EHRSHOT.

Manuscript revision. Section 3.3 now states how timing is encoded in each token and how it reaches the frozen TFM.

AuthorsOpenReview ↗

Response to Reviewer wv6w (2/5)

W2-a. Component attribution

W2: “The overall method has many components, but it is not clear to me what actually drives the performance. Critical ablations are missing.”

We examined the components one at a time. The table lists what changes on EHRSHOT when each component is replaced, with the frozen TFM kept fixed.

ComparisonWhat changesEHRSHOT AUROC / AUPRC
MEDS-Tab + TFM → BiomedBERT + TFMstatistical features → patient features learned from events, with concept-name embeddings.746/.432 → .779/.454
BiomedBERT + TFM → TabEHRconcept-name embeddings → TabEHR concept embeddings.779/.454 → .804/.510
TabEHR → random vectors as anchorsclinical meaning of the reference points.811/.525 → .756/.391
TabEHR → without the similarity termconcept–anchor similarity in the attention score.811/.525 → .679/.281
TabEHR → without the learned termlearned term in the attention score.811/.525 → .814/.521
TabEHR → recency selectioncoverage-first token selection.811/.525 → .805/.496
TabEHR → without recency encodingelapsed time of each token.811/.525 → .788/.495

Nine-task macro scores. The first two comparisons are three-seed means from Table 1; the others use one training seed (100). For reference, the between-seed SD of TabEHR on EHRSHOT is .015/.027.

Three results follow. First, learning patient features from events improves transfer over statistical features even with public concept-name embeddings (+.033/+.022), and the TabEHR concept embeddings add a further +.025/+.056. Second, within the aggregation, the gain depends on clinically meaningful anchors and on the concept–anchor similarity term: removing either lowers external AUROC by .055 and .132. Third, the learned term in the attention score is not required for this gain. Among the event inputs, removing recency encoding lowers external AUROC/AUPRC by .023/.030 (W1), and replacing coverage-first with recency selection lowers them by .006/.029. The dependence on the TFM backbone is addressed in Q3.

Manuscript revision. Section 4.3 now reports the aggregation controls, and Section 3.4 points to them and to the sensitivity to the number of anchors in Appendix I.4.

AuthorsOpenReview ↗

Response to Reviewer wv6w (3/5)

W2-b. Additional comparators

W2: “Likewise, other EHR foundation models such as CLMBR / Mamba-16k (https://arxiv.org/abs/2307.02028, https://arxiv.org/abs/2412.16178) or LLM based approaches (e.g. https://arxiv.org/abs/2502.17403v6) are not being evaluated. E.g. both Mamba-16k and Qwen3-Embedding-8B seem to perform at least as good as TabEHR in the few shot experiment in table 2.”

We added the comparators you list: Qwen3-Embedding-8B and CLMBR-T-base evaluated with the same frozen TFM, and the published Mamba-16k results, at the five support sizes of the general response. Table A compares models without target-institution pretraining on the fifteen tasks; Table B compares TabEHR with models pretrained on EHRs of the target institution.

Table A. Few-shot transfer to an external institution with a shared TFM (Overall, 15 tasks).

Patient representationk=8k=16k=32k=64k=128
Count.589.616.647.671.693
MedRep + BEHRT (FT).597 ± .004.622 ± .002.639 ± .003.651 ± .001.665 ± .002
MedTok + BEHRT (FT).596 ± .010.628 ± .008.647 ± .007.668 ± .009.686 ± .004
Qwen3-Embedding-8B.643.667.683.701.721
TabEHR.589 ± .013.643 ± .022.698 ± .025.733 ± .018.750 ± .011

Macro AUROC over five support replicates with the same frozen TFM. Fine-tuned (FT) models and TabEHR report mean ± sample SD across three source-training seeds. Qwen3-Embedding-8B is used without additional EHR training. CXR averages 14 findings as one task. Bold: highest mean within each k.

As you observed, Qwen3-Embedding-8B is higher at small support sizes (k=8 and k=16). From k=32 TabEHR is higher over all tasks, and at k=128 it reaches .750/.324 in AUROC/AUPRC against .721/.285. From k=64, TabEHR is higher than Qwen3-Embedding-8B in every task group on both metrics.

Table B. Comparison with models pretrained on target-institution EHRs.

ModelTarget-institution EHR pretrainingPredictork=8k=16k=32k=64k=128
CLMBR-T-baseOTFM.617.643.670.685.717
Mamba-16k (reported)OLR≈.654≈.681≈.714≈.738≈.766
TabEHRXTFM.591 ± .014.647 ± .023.704 ± .027.740 ± .019.773 ± .011

All rows are re-aggregated to the task sets of the Mamba report: 14 tasks without CXR at k=8–64, and 13 tasks, also without Celiac, at k=128. Mamba values are reconstructed from published task-group means (Context Clues, Appendix G and Tables 11–13). Mamba + LR uses k validation labels per class in addition to k training labels; the TFM rows use k labels per class. Bold: highest value within each k.

CLMBR-T-base and Mamba-16k are pretrained on EHRs of the target institution, and TabEHR is not. With the same frozen TFM, TabEHR is similar to CLMBR-T-base at k=16 and higher from k=32. At k=128 on this task set, TabEHR has the highest AUROC and AUPRC among the directly evaluated representations (Appendix J.5): .773/.359, against .736/.316 for Qwen3-Embedding-8B and .717/.283 for CLMBR-T-base. The published Mamba-16k values are higher at k=8–32; at k=64 and k=128, TabEHR reaches the same level (.740 and .773 against ≈.738 and ≈.766) with half as many target labels per class.

Manuscript revision. Appendix J.5 now reports the task-aligned comparison at k=16 and k=128 in AUROC and AUPRC, and marks which models are pretrained on target-institution EHRs and which values are taken from the literature.

Reviewer_7cnKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TabEHR, a method that converts longitudinal patient records into shared feature columns for a frozen tabular foundation model. It builds medical concept embeddings from ontology-informed descriptions generated by an LLM, then clusters these embeddings to create semantic anchors. The anchors aggregate each patient’s events into column-specific representations that incorporate clinical concepts, numerical values, and timing. The event encoder and aggregation module are trained through the frozen model’s prediction loss. At inference, labeled support patients provide context for predicting query patients without further parameter updates. Experiments on MIMIC-IV and EHRSHOT show improved average performance over the evaluated baselines, with larger gains on the external cohort.

Critical strengths

Learning to convert irregular patient histories into inputs for a frozen tabular foundation model is an interesting idea. Training this interface through the prediction loss allows it to preserve information that is useful for the clinical tasks.

Shared semantic anchors provide corresponding columns across patients, even when hospitals use different codes for related clinical concepts. This is a useful way to address differences in recorded vocabularies.

The same model can make predictions for different tasks by changing the labeled support examples, without further parameter updates. This offers flexibility compared with training a separate predictor for each outcome.

The external evaluation strengthens the paper. The model improves average performance on EHRSHOT using labeled MIMIC-IV patients as support, showing that the learned representation can transfer beyond the training institution.

Critical weaknesses

The paper does not clearly establish the benefit of semantic anchors over simpler alternatives. Comparisons with random fixed anchors and learned attention queries, using the same architecture and training setup, would help show whether their clinical meaning matters.

Appendix E.5 describes inconsistencies in processing duplicate values and their units. The authors should fix this issue and rerun the main experiments to check whether it affects the reported results.

The external results are promising, but the gains vary across tasks, while the internal improvement over statistical features with the same TFM is small. Paired confidence intervals would help show how reliable these differences are.

The few-shot comparisons involve representations trained with different levels of supervision. Including a sequence encoder trained on the same source tasks would make the comparison more informative. These experiments should also be repeated across source-training seeds.

Questions for authors

See the weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7cnK (1/2)

We thank Reviewer 7cnK for the constructive comments. General responses C2, C4, and C3 give the full results for W1, W3, and W4; we answer each point below.

W1. Benefit of semantic anchors over simpler alternatives

W1: “The paper does not clearly establish the benefit of semantic anchors over simpler alternatives. Comparisons with random fixed anchors and learned attention queries, using the same architecture and training setup, would help show whether their clinical meaning matters.”

We ran both comparisons you suggested, with the same architecture and training setup (Table C2-T1 in the general response). The clinical meaning of the anchors matters: replacing the semantic anchors with fixed random vectors lowers EHRSHOT AUROC/AUPRC by .055/.134, several times the between-seed SD of TabEHR (.015/.027).

Random fixed anchors. A fixed set of reference points is not sufficient; the points must be clinically meaningful. Replacing the semantic anchors with fixed random vectors lowers EHRSHOT AUROC/AUPRC from .811/.525 to .756/.391 (rows 1–2). Keeping the semantic anchors but computing each concept's similarity with another concept's embedding lowers it further, to .704/.323 (row 3).

Learned attention queries. Learned queries do not replace the anchors. Pooling with learned queries reaches .668/.267 (row 7), and adding self-attention over events gives .676/.267 (row 4). The difference comes from the concept–anchor similarity term: semantic queries without that term are at the same level (.679/.281, row 6), while the similarity term alone stays within .004 of TabEHR on EHRSHOT (.814/.521, row 5). The benefit of the anchors therefore does not require the learned term.

Manuscript revision. We added the main controls to Section 4.3 and the full table with experimental conditions to Appendix I.5, and linked them from Section 3.4.

W2. Value-processing inconsistency (Appendix E.5)

W2: “Appendix E.5 describes inconsistencies in processing duplicate values and their units. The authors should fix this issue and rerun the main experiments to check whether it affects the reported results.”

We fixed the issue and reran the main experiments. The reported results are not affected: retraining TabEHR with joint value–unit handling for same-time duplicate records gives .898/.562 on MIMIC-IV and .810/.522 on EHRSHOT, against .899/.558 and .804/.510 in the submission (three seeds each).

The fix. Previously, values from duplicate records of the same concept at the same timestamp could be aggregated while the unit label was selected independently, so a value could be paired with another record's unit. Joint value–unit handling first selects one unit group according to the unit-handling rules and then takes the median of the measurements within that group. It changes 0.07–0.09% of event tokens, concentrated in one measurement per cohort: body height in MIMIC-IV (93% of changed tokens) and the total cholesterol/HDL ratio in EHRSHOT (98%).

Effect on the results. Two checks agree. Evaluating the submitted checkpoints on inputs built with joint handling changes macro AUROC and AUPRC by less than .001 in both cohorts. Retraining with three seeds changes them by less than .001 and +.003 on MIMIC-IV and by +.005 and +.012 on EHRSHOT, within the between-seed SD of the retrained model (.006/.012 and .012/.027). Because the results are unchanged, the main tables keep the submitted values.

Manuscript revision. Appendix E.5 now describes joint value–unit handling and reports both sets of results.

W3. Variation across tasks and paired confidence intervals

W3: “The external results are promising, but the gains vary across tasks, while the internal improvement over statistical features with the same TFM is small. Paired confidence intervals would help show how reliable these differences are.”

We computed the patient-level paired intervals you asked for (general response C4). On EHRSHOT, TabEHR is higher than MEDS-Tab + TFM by +.0587 [.0520, .0658] in macro AUROC and +.0779 [.0671, .0880] in macro AUPRC. On MIMIC-IV the two methods are similar (both intervals include zero), so the advantage is in transfer to another institution. As you note, the gain varies across tasks. It is largest on the tasks where statistical features transfer least (Sepsis, MI, and Fx), and it does not depend on a single task: leaving out any one of the nine keeps both intervals above zero.

Manuscript revision. Section 4.1 now states that both intervals exclude zero, and Appendix I.1 reports the intervals and the bootstrap procedure.

AuthorsOpenReview ↗

Response to Reviewer 7cnK (2/2)

W4. Few-shot comparison: supervision level and source-training seeds

W4: “The few-shot comparisons involve representations trained with different levels of supervision. Including a sequence encoder trained on the same source tasks would make the comparison more informative. These experiments should also be repeated across source-training seeds.”

We ran both (Tables C3-T1 and C3-T2 in the general response). With source supervision matched and three source-training seeds, TabEHR is higher than both fine-tuned models from k=32 in every seed, and at k=128 it is highest in every task group in both AUROC and AUPRC.

Models trained on the same source tasks. We fine-tuned MedRep + BEHRT and MedTok + BEHRT on the same nine MIMIC-IV tasks, with the same patient splits and checkpoint selection, and evaluated them with the same frozen TFM. As you anticipated, source supervision matters: it raises the Overall AUROC of both models by .06–.08. With supervision matched, at k=128 neither fine-tuned model (.665, .686) exceeds Count (.693), which uses the benchmark's count features without source training, while TabEHR reaches .750. At k=8 the methods are close (.589–.597), and at k=16 TabEHR has the highest mean over all tasks (.643). From k=32, TabEHR exceeds the stronger fine-tuned model by .051–.065 on all tasks and by .064–.076 on Unseen outcomes (Table C3-T2).

Source-training seeds. All source-supervised rows are repeated over three seeds. From k=32, TabEHR's lowest seed is above both fine-tuned models (at k=32: .671 against .639 ± .003 and .647 ± .007). At k=128, its lowest seed (.737) is above the highest value of any other row (.693). The submission reported one checkpoint at k=16 (.659); the three-seed mean is .643.

Manuscript revision. Table 2 now reports AUROC and AUPRC at k=16 and k=128 with the fine-tuned models as comparators, Figure 2 plots k=8–128, and Appendix J.3 reports every support size with standard deviations across source-training seeds and TabEHR's three seeds separately.

We hope these responses and revisions address your concerns, and we would be glad to clarify any remaining point during the discussion.

Thanks for your rebuttal. You’ve addressed my concerns, and I’m keeping my score at weak accept, which I believe is a proper assessment of the paper.

Reviewer_Ayu5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed TabEHR, a method that attempts to apply tabular foundation models for longitudinal EHR outcome predictions. The authors argued a few existing gaps in current EHR prediction approaches: EHR time series models can capture patient's sequential relationships but lack the capabilities to translate learned clinical concepts to other institutions; while other works that focused on capturing clinical concepts' meanings through ontologies and text embeddings suffer when transferred to other clinical outcome predictions due to being trained as task-specific predictors. To this end, the paper designed the framework TabEHR with 3 main components: 1. clinical concept embedding informed with medical ontological information, augmented with language models' descriptions; trained with Masked Language Modelling on the generated descriptions and Contrastive Learning between description pairs 2. Encoding patient's irregular multimodal time events by daily binning and combining clinical concept embeddings with numerical values and temporal information 3. Constructing semantic anchors to be used for contextual pooling through clustering clinical concepts in different clinical domains. The final methods demonstrated competitive performance across the two EHR datasets MIMIC-IV and EHRSHOT across nine predictive tasks, while demonstrating improved transferability across institutions.

Critical strengths

Technically sound framework: The paper’s proposed methodology is sound and logical. Using LLM generated descriptions augmented with medical ontology are efficient in design, and the semantic anchor is a well-designed component for efficient contextual pooling.

Critical weaknesses
  1. Unconvincing Improvements in Experimental Results: While the experimental results presented by the paper are competitive, it does not offer a clear edge against other established methods which are significantly less complex with fewer components like MedRep, BioMedBert. Across the 9 prediction tasks, the only ones where TabEHR achieved best results were Pheno, FX, Sepsis for MIMIC-IV only, and MI for EHRShot only. If these four tasks were not included, the AUROC advantage over MEDS-Tab + TFM would be minimal. The source of these small improvements are also unclear, since the only conclusion I can make of this table is that the paper's LLM-based encoding is better than other clinical embedding methods due to poor ablation studies (See Weakness 2).

  2. Limited Ablation Studies and Analysis: The proposed TabEHR method is a complex method with many interacting components. However, the work provided little ablation experiments to demonstrate and justify the necessity of these components. Without these ablations, it is impossible to tell which components drive the reported gains, so the method's added complexity is not justified by the evidence presented. This includes: Section 3.2 Are LLMs generated descriptions’ embeddings more powerful than raw text embeddings of the concept's names and information? Necessity of MLM and Contrastive Learning Section 3.3 Effect of temporal binning (Eq8): How the models behave in coarser or denser grids, or if temporal binning actually improves performance at all

Section 3.4 Effect of using Semantic Anchors as attention query over standard design choices like Self-attention. Effect of number of anchors \textbf{M} Effect of clustering with and without four separate domains

Questions for authors

Q1 Additional ablation results and analysis, elaborated in the Weaknesses section Q2 Do the results hold for other Tabular Foundation Models? All experiments are conducted on EXAONE as the Tabular Foundation Model, which is a fairly new model in the existing literature. I request experimental results for the method on more established Tabular Foundation Models like TabPFN [1] or TabICLv2 [2] Q3 Computational Efficiency of TabEHR: TabEHR consists of a large number of components compared with other existing methods. What's TabEHR inference speed and memory requirements compared with other methods? Q4 Clearer explanations and justifications needed for design choices like Coverage Selection (Equation 8) (what is and why coverage-first?), Value-channel vector’s asinh usage(Equation 9). Q5 More clarity on multiple implementation and reproducibility details: The authors do not provide explanation of the OMOP model they utilized, which is unfamiliar for most ML researchers. As I understand, it's a standardized paradigm to store clinical definitions. The LLMs used to generate clinical descriptions are never specified, as well as the generation settings like temperature. More implementation details are needed for reproducibility.

[1] Hollmann, N., Müller, S., Purucker, L. et al. Accurate predictions on small data with a tabular foundation model. Nature 637, 319–326 (2025). https://doi.org/10.1038/s41586-024-08328-6 [2] Qu, Jingang, et al. "TabICLv2: A better, faster, scalable, and open tabular foundation model." arXiv preprint arXiv:2602.11139 (2026).

5 comments
AuthorsOpenReview ↗

Response to Reviewer Ayu5 (1/5)

We thank Reviewer Ayu5 for the detailed comments. General responses C4, C2, and C1 give the full results for W1 (per-task differences), W2/Q1-d (anchors versus self-attention), and Q2 (other TFM backbones); we answer each point below.

W1. Unconvincing improvements

W1: “…it does not offer a clear edge against other established methods which are significantly less complex with fewer components like MedRep, BioMedBert. … If these four tasks were not included, the AUROC advantage over MEDS-Tab + TFM would be minimal. The source of these small improvements are also unclear…”

TabEHR's advantage is in transfer to another institution. It is similar to MEDS-Tab + TFM at the source institution and higher on EHRSHOT by +.0587/+.0779 in macro AUROC/AUPRC, with paired 95% intervals above zero. As you observe, the gain is concentrated in Pheno, Fx, Sepsis, and MI, and without them the two methods are similar in macro AUROC. These are the tasks where statistical features transfer least, and the gain does not depend on any single one of them (general response C4).

MedRep and BiomedBERT. MedRep and BiomedBERT provide concept embeddings; predicting from them still requires a patient-level model. Table 1 uses each embedding in two such models. Within EHR sequence models (+ BEHRT), which require EHR pretraining and a separately fine-tuned model for each task, MedRep, BiomedBERT, and TabEHR concept embeddings reach .731/.385, .762/.423, and .756/.397 on EHRSHOT. Within TabEHR's pipeline (+ TFM), the same three embeddings reach .793/.486, .779/.454, and .804/.510, all above MEDS-Tab + TFM (.746/.432). This pipeline trains one module of 508,800 parameters for all nine tasks and runs only that module and the frozen TFM at inference. The transfer gain therefore holds with all three embeddings, and the TabEHR concept embeddings give the highest result of the three: above BiomedBERT + TFM in every seed, and above MedRep + TFM by .011/.024. W2/Q1 examines the individual components.

W2 / Q1-a. Generated descriptions

W2 / Q1: “Section 3.2 Are LLMs generated descriptions’ embeddings more powerful than raw text embeddings of the concept's names and information?”

Yes, in our pipeline. Replacing BiomedBERT embeddings of OMOP concept names with embeddings learned from the generated clinical descriptions raises nine-task macro AUROC/AUPRC by .009/.015 on MIMIC-IV and by .025/.056 on EHRSHOT, and TabEHR is higher than BiomedBERT + TFM in each of the three seeds.

Concept representationMIMIC-IV AUROC / AUPRCEHRSHOT AUROC / AUPRC
BiomedBERT + TFM (concept names encoded by BiomedBERT).890 / .543.779 / .454
TabEHR (embeddings learned from generated clinical descriptions).899 / .558.804 / .510

Nine-task macro averages over three training seeds. Both rows use the same event inputs, aggregation architecture, and frozen TFM; the anchors are rebuilt and the patient-level module is retrained for each representation. Bold: best in each column.

A concept name is a short terminology label. The generated clinical description states the clinical context of the concept, using its name and typed ontology relations, and the concept encoder is trained on these descriptions. The comparison changes the text and the encoder training together, so it evaluates description-based representation learning as a whole.

Manuscript revision. Appendix D.3 describes how the concept-name embeddings are built, and Section 4.1 relates this comparison to the main results.

AuthorsOpenReview ↗

Response to Reviewer Ayu5 (2/5)

W2 / Q1-b. MLM and contrastive learning

W2 / Q1: “…Necessity of MLM and Contrastive Learning”

The concept encoder is trained in two stages (Section 3.2). We first adapt it to the generated clinical descriptions with MLM, and then combine MLM with a concept-level contrastive objective that places the integrated synthesis and the role-specific descriptions of the same concept nearby while distinguishing different concepts.

To evaluate the contrastive stage, we took the encoder at the end of the MLM stage, embedded the same generated descriptions with it, and retrained TabEHR on these concept embeddings.

Concept encoderMIMIC-IV AUROC / AUPRCEHRSHOT AUROC / AUPRC
MLM only.892 ± .001 / .555 ± .002.797 ± .002 / .495 ± .001
MLM, then MLM + contrastive (TabEHR).899 ± .003 / .558 ± .004.804 ± .015 / .510 ± .027

Nine-task macro averages ± between-seed SD over three training seeds. Both rows embed the same generated descriptions and use the same event inputs, aggregation architecture, and frozen TFM; for each encoder, the anchors are rebuilt and the event encoder and anchor aggregator are retrained. Bold: best in each column.

Adding the contrastive stage raises all four metrics, by .007/.003 on MIMIC-IV and by .007/.015 on EHRSHOT. Already after the MLM stage, the embeddings are higher than BiomedBERT embeddings of concept names on EHRSHOT (.797/.495 against .779/.454; W2/Q1-a).

Manuscript revision. Section 3.2 now states the role of each objective.

W2 / Q1-c. Temporal binning and resolution

W2 / Q1: “Section 3.3 Effect of temporal binning (Eq8): How the models behave in coarser or denser grids, or if temporal binning actually improves performance at all”

Temporal binning compresses repeated observations before applying the fixed event-token budget. Observations of the same concept within each time bin are summarized into one token, using narrower bins for recent observations and wider bins for older observations. Increasing δ produces coarser bins and greater compression.

The table below shows how this changes the number of tokens presented to the subsequent selection stage.

Temporal binningMIMIC-IV median tokensMIMIC-IV >512 tokensEHRSHOT median tokensEHRSHOT >512 tokens
Daily19824.7%48348.6%
δ = 0.2517116.5%33936.5%
δ = 0.516413.4%30831.5%
δ = 1.015610.6%28126.3%

Counted on the inputs used in the paper, pooling prediction instances across nine tasks (MIMIC-IV test instances and EHRSHOT instances). Counts are measured before token selection; these are pooled instance statistics, not task-macro averages. At δ = 0.5 the per-task EHRSHOT rates are those of Table 12 (27.4–39.1%).

At δ=0.5, median token counts decrease from 198 to 164 in MIMIC-IV and from 483 to 308 in EHRSHOT relative to daily binning. The proportion exceeding the 512-token budget falls from 24.7% to 13.4% and from 48.6% to 31.5%, respectively. Binning therefore reduces how often token selection is required, at the cost of summarizing more observations within each bin.

Varying the input resolution for the same model while maintaining a 512-token budget, all evaluated elapsed-time configurations yielded higher AUROC and AUPRC than daily binning in both cohorts, with comparatively small differences among the elapsed-time settings. At δ=0.5, the AUROC/AUPRC increases were 0.43/0.86 percentage points in MIMIC-IV and 1.21/3.73 percentage points in EHRSHOT.

Elapsed-time configurations retain finer detail for recent observations while summarizing older observations through their counts, numerical values, and most recent observation times. This may explain the small differences among the elapsed-time settings.

These results support the utility of elapsed-time binning relative to daily binning under this evaluation.

Manuscript revision. Section 3.3 now reports the reduction in instances exceeding the token budget, and Appendix E.2 adds the token counts at each resolution.

AuthorsOpenReview ↗

Response to Reviewer Ayu5 (3/5)

W2 / Q1-d. Semantic anchors versus self-attention

W2 / Q1: “Section 3.4 Effect of using Semantic Anchors as attention query over standard design choices like Self-attention.”

We ran the self-attention alternative you asked about (Table C2-T1 in the general response). One self-attention layer over a patient's events, followed by pooling with 512 learned queries, reaches .676/.267 in EHRSHOT AUROC/AUPRC, against .811/.525 for TabEHR with the same event inputs, frozen TFM, and number of output cells (rows 4 and 1). Learned queries without the self-attention layer are similar (.668/.267, row 7).

The difference comes from the concept–anchor similarity term, which a design with learned queries does not have. Without that term, semantic queries are at the same level as learned queries (.679/.281, row 6); with that term alone, performance stays within .004 of TabEHR on EHRSHOT (.814/.521, row 5). The term also requires clinically meaningful anchors: with fixed random vectors in their place, performance falls to .756/.391 (row 2).

Manuscript revision. We added these controls to Section 4.3 and the full table with experimental conditions to Appendix I.5.

W2 / Q1-e. Number of anchors

W2 / Q1: “Effect of number of anchors M”

Most of the external improvement is obtained by 512 anchors. Increasing M from 256 to 512 raises EHRSHOT AUROC/AUPRC by .053/.093; doubling it again to 1,024 adds .002/.018, and MIMIC-IV scores remain close across the three settings.

Anchors MMIMIC-IV AUROC / AUPRCEHRSHOT AUROC / AUPRC
256.898 / .564.765 / .447
512.895 / .554.818 / .540
1,024.897 / .565.820 / .558

Nine-task macro scores with one training seed (100), with joint value–unit handling for same-time duplicate records (Appendix E.5). All settings keep the domain-wise weighted Ward construction; checkpoints are selected by MIMIC-IV validation macro AUROC, without EHRSHOT labels.

We keep M=512 as a practical compromise between prediction performance and representation size: it obtains most of the improvement with half as many output cells as M=1,024.

Manuscript revision. Section 3.4 links to Appendix I.4, which reports this comparison and how the number of anchors changes concept coverage and assignment patterns across institutions.

W2 / Q1-f. Domain-wise clustering

W2 / Q1: “Effect of clustering with and without four separate domains”

Separating the four domains has two effects, which we tested separately: it fixes domain boundaries, and it gives each domain the same number of anchors. The boundaries have little effect on performance; the equal allocation matters.

Anchor construction (M=512)MIMIC-IV AUROC / AUPRCEHRSHOT AUROC / AUPRC
Separate domains, 128 anchors per domain (TabEHR).895 / .554.818 / .540
Joint clustering across domains.897 / .559.811 / .523
Separate domains, anchors allocated by source-event frequency.890 / .538.770 / .430

Nine-task macro scores with one training seed (100), with joint value–unit handling as in W2/Q1-e; checkpoints are selected by MIMIC-IV validation macro AUROC. Bold: best in each column.

Domain boundaries. Joint clustering gives external performance close to the domain-wise construction (.811/.523 against .818/.540); the difference is within the between-seed SD of TabEHR on EHRSHOT (.015/.027). The concept embedding space already separates the domains: 82.2% of the jointly formed clusters contain concepts from a single domain.

Allocation across domains. Allocating the 512 anchors in proportion to source-event frequency assigns 38, 73, 369, and 32 anchors to Condition, Drug, Measurement, and Procedure, and lowers EHRSHOT AUROC/AUPRC by .048/.110. Equal allocation gives each domain its own reference points regardless of how often its events are recorded at the source institution, and this is the property of the domain-wise design that affects transfer.

Manuscript revision. Appendix I.4 reports this comparison (Table 33).

Q2. Transfer to other TFM backbones

Q2: “Do the results hold for other Tabular Foundation Models?”

We trained and evaluated TabEHR with both requested backbones, TabICLv2 and TabPFN-3.5, over three training seeds (general response C1). TabEHR can be trained with all three frozen backbones. Its gain over statistical features appears with EXAONE Tabular, where it gives the highest EHRSHOT result among the six backbone–input combinations; with TabICLv2 and TabPFN-3.5, statistical features are higher. The general response relates this pattern to how each backbone processes the anchor columns.

Comments on the paper

5 comments
AuthorsOpenReview ↗

G1 — General response (1/5): strengths and summary of revisions

We thank the reviewers for their careful reading and constructive comments. TabEHR maps irregular longitudinal EHRs into shared semantic columns for a frozen tabular foundation model (TFM); it is trained once on MIMIC-IV and applied to EHRSHOT without parameter updates or EHRSHOT labels.

The reviewers found the framework technically sound, with LLM-generated, ontology-informed concept descriptions and semantic anchors as well-designed components (Ayu5). They found the derivation of semantic embeddings for OMOP concepts novel, on a question that remains open: how EHRs can be prepared for foundation models in a way that generalizes across healthcare systems (wv6w). They also noted that shared semantic anchors give patients corresponding columns across institutions with different recorded vocabularies, that one model predicts different tasks by changing the labeled support examples, and that the external evaluation strengthens the paper (7cnK).

The revised manuscript marks new and changed text in blue. The main changes are:

  • Aggregation controls (7cnK, Ayu5, wv6w; C2). Random anchors, mismatched concept embeddings, learned queries with self-attention, and removal of the similarity term or the learned term, with the same architecture and training (Section 4.3, Table 3; Appendix I.5).
  • Other TFM backbones (Ayu5, wv6w; C1). TabEHR trained with frozen TabICLv2 and TabPFN-3.5, three source-training seeds each (Appendix I.6; Discussion).
  • Uncertainty of the main comparison (Ayu5, 7cnK; C4). Paired bootstrap intervals and the difference with each task left out (Section 4.1; Appendix I.1).
  • Few-shot evaluation (7cnK, wv6w; C3). Support sizes k=8–128 in AUROC and AUPRC, with EHR sequence models fine-tuned on the same source tasks and three source-training seeds (Section 4.2, Table 2, Figure 2; Appendix J).
  • Same-time duplicate records (7cnK). Joint value–unit handling, with TabEHR retrained over three seeds (Appendix E.5).
  • Further analyses (Ayu5, wv6w). Concept retrieval after removing candidates linked by a supplied relation (Section 4.3; Appendix G.4), concept-name embeddings (Appendix D.3), temporal grid resolution (Appendix E.2), anchor allocation across clinical domains (Appendix I.4), and inference cost (Appendix I.7).
  • Clarifications (Ayu5, wv6w). Description generation and the roles of MLM and the contrastive objective (Section 3.2; Appendix B.1), how timing is encoded (Section 3.3), and the scope of the evaluation (Discussion).

We answer four concerns raised by more than one reviewer below (C1–C4) and the remaining points in the individual responses.

AuthorsOpenReview ↗

G2 — General response (2/5): C1. Transfer across TFM backbones (Ayu5 Q2 / wv6w Q3)

Ayu5 Q2: "Do the results hold for other Tabular Foundation Models? … I request experimental results for the method on more established Tabular Foundation Models like TabPFN [1] or TabICLv2 [2]"

wv6w Q3: "Have other pretrained TFMs been tested, or is the approach specific to EXAONE Tabular?"

We trained and evaluated TabEHR with TabICLv2 and TabPFN-3.5 in addition to EXAONE Tabular, with three training seeds for each backbone (Table C1-T1). TabEHR can be trained with all three frozen backbones, and its gain over statistical features depends on the backbone.

Table C1-T1. Statistical features and TabEHR cells with three frozen TFM backbones.

BackboneInputMIMIC-IV AUROC / AUPRCEHRSHOT AUROC / AUPRC
TabICLv2Statistical features.879 / .537.750 / .430
TabICLv2TabEHR cells.8767 ± .0024 / .4888 ± .0044.7239 ± .0052 / .3545 ± .0104
TabPFN-3.5Statistical features.885 / .551.752 / .415
TabPFN-3.5TabEHR cells.8494 ± .0042 / .4069 ± .0071.6507 ± .0028 / .2454 ± .0039
EXAONE TabularStatistical features.898 / .549.746 / .432
EXAONE TabularTabEHR cells.899 ± .003 / .558 ± .004.804 ± .015 / .510 ± .027

Nine-task macro AUROC / AUPRC under the evaluation protocol in Section 4.1. Both cohorts use labeled MIMIC-IV training instances as support. TabEHR cell results are mean ± sample SD across three training seeds; each backbone is frozen while the event encoder and anchor aggregator are trained. Statistical features are MEDS-Tab-style summaries; TabEHR cells are learned vector representations formed using shared semantic anchors. Input construction and cell interfaces are detailed in Appendix I.6. Bold: best in each column.

With EXAONE Tabular, TabEHR is similar to statistical features on MIMIC-IV (.899/.558 against .898/.549) and higher on EHRSHOT (.804/.510 against .746/.432). This is the highest EHRSHOT result among the six combinations in the table; statistical features reach .746–.752 AUROC on EHRSHOT with every backbone, so the difference is not explained by a stronger backbone for tabular inputs in general. With TabICLv2 and TabPFN-3.5, statistical features are higher than TabEHR cells in both cohorts.

One architectural difference is consistent with this pattern. EXAONE Tabular keeps a separate representation for each anchor column while it attends across features and across patients along the same column. TabICLv2 and TabPFN-3.5 combine a patient's features into one row representation before in-context prediction, so the column correspondence that the shared anchors establish across patients is used less directly. This is an interpretation of the three results, not a controlled comparison of architectures.

Manuscript revision. We added the three-seed comparison and this interpretation to Appendix I.6, and the Discussion now states that the benefit of learned EHR representations depends on how the TFM processes them.

AuthorsOpenReview ↗

G3 — General response (3/5): C2. Anchor and attention-score controls (7cnK W1 / Ayu5 Q1-d)

7cnK W1: "The paper does not clearly establish the benefit of semantic anchors over simpler alternatives. Comparisons with random fixed anchors and learned attention queries, using the same architecture and training setup, would help show whether their clinical meaning matters."

Ayu5 W2 / Q1: "…Effect of using Semantic Anchors as attention query over standard design choices like Self-attention."

The clinical meaning of the anchors matters, and its effect comes through the concept–anchor similarity term. Table C2-T1 collects the controls on anchor construction and attention scoring that the two reviewers requested; all rows keep the event inputs, the frozen TFM, and 512 output cells.

Table C2-T1. Anchor and attention-score controls.

#Anchors / queriesAttention score termsMIMIC-IV AUROC / AUPRCEHRSHOT AUROC / AUPRC
1Fixed semantic anchors (TabEHR)Similarity + learned.900 / .562.811 / .525
2Fixed random vectorsSimilarity + learned.878 / .503.756 / .391
3Fixed semantic anchors, mismatched similaritySimilarity + learned.879 / .498.704 / .323
4Learned queries + self-attention over eventsLearned only.868 / .419.676 / .267
5Fixed semantic anchorsSimilarity only.896 / .548.814 / .521
6Fixed semantic anchorsLearned only.862 / .436.679 / .281
7Learned queriesLearned only.847 / .431.668 / .267
8Fixed random vectorsLearned only.849 / .408.663 / .264

Nine-task macro AUROC / AUPRC with one training seed (100) per row. Similarity: the concept–anchor similarity term; learned: the learned term of the attention score. Mismatched: each concept's similarity is computed with another concept's embedding. For reference, the between-seed SD of TabEHR on EHRSHOT is .015/.027. Bold: best in each column.

Clinically meaningful reference points are required. With both score terms kept, replacing the semantic anchors with fixed random vectors lowers EHRSHOT AUROC/AUPRC from .811/.525 to .756/.391, and computing each concept's similarity with another concept's embedding lowers it to .704/.323 (rows 1–3). These decreases (.055/.134 and .107/.202) are several times the between-seed SD.

The gain comes from the similarity term. With that term alone, performance stays within .004 of TabEHR on EHRSHOT (.814/.521, row 5). Without it, EHRSHOT AUROC is between .663 and .679 whether the queries are semantic, learned, or random, with or without self-attention over events (rows 4 and 6–8). The benefit therefore does not require the learned term.

Manuscript revision. We added the main controls to Section 4.3 (Table 3) and the full table with experimental conditions to Appendix I.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.