- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a controlled analysis of how input representations (input embeddings) and pretraining objectives (pretraining losses) affect the zero-shot performance of generative EHR foundation models. The authors compare three model families drawn from prior work. The ETHOS-style Discretized model bins both time intervals and numerical values and is trained with cross-entropy; the multivariateGPT-style Continuous-fused model feeds normal-score-transformed values through xVal-like embeddings and is trained with a Gaussian NLL; and the Delphi-2M-style MTPP model represents time with a homogeneous Poisson process, modified to handle concurrent events through a Bernoulli gate. All three share the same tokenized sequence layout, a GPT-2 backbone of roughly 40M parameters, and a 30k-step training budget. For evaluation, the authors also propose the ZEHRO benchmark, which covers diagnosis classification, next-event time prediction, and lab-value regression under both one-step prediction and long-horizon simulation with 64 trajectories, applied to MIMIC-IV and a private institutional dataset. They further run a 2×2 factorial ablation crossing the numerical input encoding (discrete/continuous) with the numerical objective (categorical/Gaussian). They conclude that Discretized is stronger for classification, Continuous-fused is stronger for regression, MTPP is strong at predicting long time intervals, and changing the pretraining objective has a larger effect on performance than changing the input embedding.
Critical strengths
- Timely problem formulation: The problem is well motivated by the current state of the field, in which existing models differ in data, tokenizer, backbone, and evaluation, making direct comparison difficult.
Critical weaknesses
W1. Insufficient support for the headline conclusion (loss > embedding) and limited value as design guidance
The headline conclusion of Section 6.2, "The pretraining loss has a larger effect than the input representation," is a central claim of the paper and appears in both the abstract and the list of contributions. However, the evidence for this claim is partly a consequence of the evaluation structure, which tends to magnify the effect of the loss, and even if the conclusion holds, it is hard to see how it provides actionable guidance for future model design.
- Relationship between the evaluation structure and the claim: Using the fully discrete model (Discretized) as the reference, the authors compare the performance change from swapping only the input embedding against swapping only the loss, citing +1.237 vs. −0.076 for one-step time NMAE ("16 times") and +0.124 vs. +0.015 for one-step regression R² ("approximately seven times") (Table 8). However, changing the loss changes the very distribution the model outputs (categorical → Gaussian), and the one-step evaluation scores exactly that output. Changing the input embedding, by contrast, only alters how the already-observed history is encoded. The one-step time and value predictions, where the large gaps appear, are therefore evaluations in which the effect of the loss is structurally bound to be large. In long-horizon classification, where this structural factor is absent, there are cases in which the embedding change has the larger effect, such as AMI (embedding −0.021 vs. loss −0.012) and MASLD (+0.052 vs. +0.035).
- Limited value as design guidance: Even if the conclusion holds, the authors themselves conclude that no single combination is optimal across all tasks (lines 474–475), and the trade-offs of each configuration are not summarized. Moreover, apart from the analysis of time-interval distributions for timing (Figs. 9 and 10), there is little mechanistic analysis of why these differences arise. To serve as practical design guidance, the claims and analyses should be organized around which configuration to choose for a given data characteristic and task type.
- Inconsistencies between the text and the tables: (i) The text states that changing the loss has a larger effect "across all evaluation tasks" (lines 419–421), but this is inconsistent with Table 8 in cases such as the AMI and MASLD classification tasks noted above. (ii) The statement on line 436, "MTPP always outperforms the ablation with the same continuous loss but discrete input embeddings," appears from context to refer to a comparison between Continuous-fused and Discrete-gaussian. Even under that reading, Discrete-gaussian performs better on AMI, MASLD, and stroke classification and on MASLD onset prediction (Table 7), so "always" does not hold.
W2. Representativeness of the compared methods and the scope of generalization
- The time loss of Continuous-fused is a single Gaussian after a normal-score transformation. As the authors themselves note, this assumption has difficulty capturing the multimodal (spiky) time-interval distribution in MIMIC. The finding that "discretization is superior for timing" may therefore reflect a property of the unimodal Gaussian head rather than of continuous approaches in general. Comparing against mixture density heads, normalizing flows (as in Draxler et al., cited in the paper), or histogram-style heads would make the conclusion more robust.
- Likewise, MTPP takes a homogeneous form (a history-conditioned constant intensity with a memoryless exponential), which is a relatively simple member of the point-process family. Concluding from this that "the point process hurt both classification and regression" seems to be a somewhat broad generalization. I recommend either including more expressive TPPs (e.g., Neural Hawkes or log-normal-mixture intensity-free models) or narrowing the scope of the conclusion.
W3. Definition and validity of the timing evaluation metric
- All time-to-event conclusions, one of the three evaluation axes, rely on NMAE, yet NMAE is never defined in the paper. Given that values reach 14.9 in the 1 min–1 h bin, it appears to be a relative error normalized by the true value. If so, it is highly sensitive to extreme values, and the rationale for aggregating the four bins by their geometric mean also needs to be explained.
- The paper also does not specify how each model produces a point estimate (sample mean, median, bin representative value, etc.). Under MAE, the median is optimal, and using the mean of an exponential distribution can be disadvantageous for short intervals, so the ranking of MTPP may depend on this choice.
- Long-horizon time-to-event is evaluated only on positives, so censoring and cases in which the event is never predicted are not taken into account. This is also inconsistent with the censoring concept depicted in Fig. 2, and reporting censoring-aware metrics such as the C-index alongside would be helpful.
W4. Absence of analysis from a fine-tuning perspective
- The authors restrict their contribution to zero-shot evaluation. However, in the medical domain, where system safety is critical, models are often deployed after fine-tuning (or at least linear probing). This is all the more relevant given that most ZEHRO tasks (mortality, readmission, LOS, ED outcomes, regression of common lab values, etc.) have abundant labels.
- Considering that the paper proposes a standard benchmark, the absence of another clinically important evaluation axis beyond zero-shot is a weakness of this work.
Questions for authors
See weaknesses above.