acceptodds
Withdrawn from ICLR 2027

Input Representations Meet Pretraining Objectives: A Controlled Study of Generative EHR Foundation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Accurately modeling irregular event timing and numerical measurements is central to zero-shot prediction with generative Electronic Health Record (EHR) foundation models. Recent work has studied the impact of input event representations and loss function separately, overlooking their correlation and joint impact. Additionally, there is no comprehensive benchmark that systematically evaluates zero-shot inference performance on a large and diverse number of tasks. To address these problems, we conduct a control study that aligns the tokenizer, backbone, and training budget to isolate the effect of input embeddings and pretraining loss across three model families: Discretized, Continuous-fused, and marked temporal point-process (MTPP). For zero-shot evaluation, we introduce ZEHRO, a comprehensive zero-shot benchmark that spans two datasets, three task types, and two time horizons. Our results show that continuous representations and loss functions benefit regression, while discretization leads in classification. The point-process approach achieves the best performance in predicting large time intervals and remains competitive in long-horizon time-to-event prediction. Our ablation studies further show that changing the pretraining objective generally has a larger effect on performance than changing the input embeddings, and that these changes can have aligned or opposing effects across different tasks. These findings reveal task-dependent trade-offs in input representations and objective choices, informing their joint design in future generative EHR foundation models.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
EHR foundation models, pretraining loss, zero-shot inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5XrjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a controlled analysis of how input representations (input embeddings) and pretraining objectives (pretraining losses) affect the zero-shot performance of generative EHR foundation models. The authors compare three model families drawn from prior work. The ETHOS-style Discretized model bins both time intervals and numerical values and is trained with cross-entropy; the multivariateGPT-style Continuous-fused model feeds normal-score-transformed values through xVal-like embeddings and is trained with a Gaussian NLL; and the Delphi-2M-style MTPP model represents time with a homogeneous Poisson process, modified to handle concurrent events through a Bernoulli gate. All three share the same tokenized sequence layout, a GPT-2 backbone of roughly 40M parameters, and a 30k-step training budget. For evaluation, the authors also propose the ZEHRO benchmark, which covers diagnosis classification, next-event time prediction, and lab-value regression under both one-step prediction and long-horizon simulation with 64 trajectories, applied to MIMIC-IV and a private institutional dataset. They further run a 2×2 factorial ablation crossing the numerical input encoding (discrete/continuous) with the numerical objective (categorical/Gaussian). They conclude that Discretized is stronger for classification, Continuous-fused is stronger for regression, MTPP is strong at predicting long time intervals, and changing the pretraining objective has a larger effect on performance than changing the input embedding.

Critical strengths
  1. Timely problem formulation: The problem is well motivated by the current state of the field, in which existing models differ in data, tokenizer, backbone, and evaluation, making direct comparison difficult.
Critical weaknesses

W1. Insufficient support for the headline conclusion (loss > embedding) and limited value as design guidance

The headline conclusion of Section 6.2, "The pretraining loss has a larger effect than the input representation," is a central claim of the paper and appears in both the abstract and the list of contributions. However, the evidence for this claim is partly a consequence of the evaluation structure, which tends to magnify the effect of the loss, and even if the conclusion holds, it is hard to see how it provides actionable guidance for future model design.

  • Relationship between the evaluation structure and the claim: Using the fully discrete model (Discretized) as the reference, the authors compare the performance change from swapping only the input embedding against swapping only the loss, citing +1.237 vs. −0.076 for one-step time NMAE ("16 times") and +0.124 vs. +0.015 for one-step regression R² ("approximately seven times") (Table 8). However, changing the loss changes the very distribution the model outputs (categorical → Gaussian), and the one-step evaluation scores exactly that output. Changing the input embedding, by contrast, only alters how the already-observed history is encoded. The one-step time and value predictions, where the large gaps appear, are therefore evaluations in which the effect of the loss is structurally bound to be large. In long-horizon classification, where this structural factor is absent, there are cases in which the embedding change has the larger effect, such as AMI (embedding −0.021 vs. loss −0.012) and MASLD (+0.052 vs. +0.035).
  • Limited value as design guidance: Even if the conclusion holds, the authors themselves conclude that no single combination is optimal across all tasks (lines 474–475), and the trade-offs of each configuration are not summarized. Moreover, apart from the analysis of time-interval distributions for timing (Figs. 9 and 10), there is little mechanistic analysis of why these differences arise. To serve as practical design guidance, the claims and analyses should be organized around which configuration to choose for a given data characteristic and task type.
  • Inconsistencies between the text and the tables: (i) The text states that changing the loss has a larger effect "across all evaluation tasks" (lines 419–421), but this is inconsistent with Table 8 in cases such as the AMI and MASLD classification tasks noted above. (ii) The statement on line 436, "MTPP always outperforms the ablation with the same continuous loss but discrete input embeddings," appears from context to refer to a comparison between Continuous-fused and Discrete-gaussian. Even under that reading, Discrete-gaussian performs better on AMI, MASLD, and stroke classification and on MASLD onset prediction (Table 7), so "always" does not hold.

W2. Representativeness of the compared methods and the scope of generalization

  • The time loss of Continuous-fused is a single Gaussian after a normal-score transformation. As the authors themselves note, this assumption has difficulty capturing the multimodal (spiky) time-interval distribution in MIMIC. The finding that "discretization is superior for timing" may therefore reflect a property of the unimodal Gaussian head rather than of continuous approaches in general. Comparing against mixture density heads, normalizing flows (as in Draxler et al., cited in the paper), or histogram-style heads would make the conclusion more robust.
  • Likewise, MTPP takes a homogeneous form (a history-conditioned constant intensity with a memoryless exponential), which is a relatively simple member of the point-process family. Concluding from this that "the point process hurt both classification and regression" seems to be a somewhat broad generalization. I recommend either including more expressive TPPs (e.g., Neural Hawkes or log-normal-mixture intensity-free models) or narrowing the scope of the conclusion.

W3. Definition and validity of the timing evaluation metric

  • All time-to-event conclusions, one of the three evaluation axes, rely on NMAE, yet NMAE is never defined in the paper. Given that values reach 14.9 in the 1 min–1 h bin, it appears to be a relative error normalized by the true value. If so, it is highly sensitive to extreme values, and the rationale for aggregating the four bins by their geometric mean also needs to be explained.
  • The paper also does not specify how each model produces a point estimate (sample mean, median, bin representative value, etc.). Under MAE, the median is optimal, and using the mean of an exponential distribution can be disadvantageous for short intervals, so the ranking of MTPP may depend on this choice.
  • Long-horizon time-to-event is evaluated only on positives, so censoring and cases in which the event is never predicted are not taken into account. This is also inconsistent with the censoring concept depicted in Fig. 2, and reporting censoring-aware metrics such as the C-index alongside would be helpful.

W4. Absence of analysis from a fine-tuning perspective

  • The authors restrict their contribution to zero-shot evaluation. However, in the medical domain, where system safety is critical, models are often deployed after fine-tuning (or at least linear probing). This is all the more relevant given that most ZEHRO tasks (mortality, readmission, LOS, ED outcomes, regression of common lab values, etc.) have abundant labels.
  • Considering that the paper proposes a standard benchmark, the absence of another clinically important evaluation axis beyond zero-shot is a weakness of this work.
Questions for authors

See weaknesses above.

Reviewer_Yxz1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This study presents the results of a controlled evaluation designed to assess how factors such as tokenization and pretraining strategies influence the performance of EHR foundation models. In doing so, the authors introduce ZEHRO, a comprehensive benchmarking platform for evaluating zero-shot performance on EHR tasks. As EHR foundation models become increasingly prevalent, there is a clear need for such a resource, making this work timely. Nevertheless, important concerns remain, particularly about the evaluative tasks chosen; e.g., it is not clear how these tasks were chosen, why the AUPRC is the appropriate metric of success, and whether the approach and tasks provide insight into how useful a EHR generative model is. Currently, the results are presented in a way that brings into question the utility of the entire approach.

Critical strengths

ZEHRO appears to be a useful resource that provides an objective pipeline for the evaluation of EHR models over a variety of different tasks.

Critical weaknesses

The methodological novelty of the work is low.

It is unclear whether the selected evaluation tasks provide meaningful insight into the performance and practical utility of a given EHR foundation model. This is a central issue that warrants substantially greater justification and discussion in the manuscript.

Do Figures 3 and 4 report macro-averaged results across all one-step and long-horizon tasks? The manuscript should clearly distinguish performance across prediction horizons and account for differences in outcome prevalence among tasks, as both factors can substantially affect the reported aggregate results.

The authors should report and discuss the prevalence of the outcomes selected for evaluating long-horizon simulations. Clinically important outcomes that one might wish to assess with a foundation model, such as mortality, can be quite rare. It is therefore unclear whether the selected tasks and metrics meaningfully characterize generative-model performance for these low-prevalence outcomes.

The manuscript should identify the 100 ICD codes used for one-step prediction and explain how they were selected.

It is unclear how prediction of 100 ICD codes over horizons ranging from one minute to one hour provides insight into the practical utility of the generative model. Which of these codes are routinely recorded at such short intervals, and what clinically meaningful use case does this evaluation represent? The authors should clarify the rationale for both the code set and the selected prediction horizons.

The authors use N = 64 rollouts to evaluate long-horizon performance, which may be insufficient for very low-prevalence outcomes. The manuscript states, “We find that for all models the performance consistently improves as we scale up N and reaches a near plateau at N = 32 or N = 64, which is consistent with findings from previous literature.” However, convergence depends in part on the prevalence of the task or token being evaluated. The authors should provide outcome-specific convergence analyses, identify the rare events included in the evaluation, report their prevalence, and demonstrate that N = 64 yields stable estimates for those events.

The authors rely exclusively on area under the precision–recall curve (AUPRC) as the evaluation metric. Although AUPRC is informative under class imbalance, reporting it without the area under the receiver operating characteristic curve (AUROC) is uncommon in medical applications and provides an incomplete view of performance. Each metric has limitations: AUPRC emphasizes performance for the positive class and is highly dependent on outcome prevalence, whereas AUROC may be unreliable for highly imbalanced outcomes. The authors should therefore report both AUROC and AUPRC, together with outcome prevalence.

Questions for authors

I am certainly willing to reconsider my score if the authors can adequately address my comments in the "Weaknesess" section.

Reviewer_AzQTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper conducts a controlled study of how input representations and pretraining objectives interact in generative EHR foundation models. The authors align the dataset, tokenized sequence layout, GPT-2–style backbone (~40M params), and training budget across three families: Discretized, Continuous-fused and MTPP. For evaluation, they propose ZEHRO, a zero-shot benchmark covering classification (AUPRC), time-to-event (NMAE across four gap bins + disease-onset timing), and lab regression (R²), each in both one-step and long-horizon-simulation forms, evaluated on MIMIC-IV and a private dataset. The paper reports the findings on both dataset.

Critical strengths
  • Genuine controlled comparison. Prior work in this space routinely confounds dataset, tokenizer, backbone, and budget. By aligning the tokenized sequence layout (so each model sees the same number of patient timelines) and sharing the backbone/optimizer/budget, the paper makes the first fair head-to-head of these three families on identical ground. This is the kind of study the subfield has been missing, and it is consequential: without it, the field has been accumulating conclusions that could just be dataset or capacity effects.

  • Crossing discrete/continuous inputs with categorical/Gaussian losses, and showing that the two factors can have opposing effects on the same task (e.g., one-step time-to-event), is more informative than the usual family-level comparison. The implication — that practitioners should pick input encoding and loss by task, not inherit them together from a reference method — is a clean, usable design guideline.

Critical weaknesses
  • The scaling law of the foundation model was not tested. All results are at ~40M params and 30k steps. Limiting the study to one capacity regime caps the generality of every finding and the benchmark the author proposed.

  • The results look inconsistent across tasks and datasets. "MTPP is best at large time intervals." True on Institution 30 d–2 y (NMAE 0.688 vs 0.780 discrete / 0.786 cont-fused). On MIMIC it is not best in that bin — Continuous-fused wins (0.581 vs MTPP 0.605). The abstract-level claim is really an Institution-only result. Short-interval timing (1 min–1 h) - MTPP is only modestly worse than Discretized on MIMIC (17.7 vs 14.9), but catastrophically worse on Institution (108.6 vs 10.1). An order-of-magnitude dataset dependence that the paper hand-waves with the "spiky vs smooth distribution" argument — plausible but post-hoc. For a benchmark, it would be helpful to create concrete criteria to evaluate rather than being hand-waving.

Questions for authors
  • For MTPP, how much of the one-step short-interval failure (NMAE 108 on Institution 1 min–1 h vs. 10 for discrete) is due to the homogeneous-Poisson assumption vs. your Bernoulli-gate modification? Did you consider a piecewise-constant or log-time rate for a cleaner MTPP comparator?

  • The "Institution" dataset is anonymized but used for a key external-validation claim. Can you say more about licensing/release, or at minimum provide the preprocessing code so results are independently reproducible?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.