- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper analyzes how the way patient trajectories are sampled and assembled into pretraining sequences affects the zero-shot clinical prediction performance of autoregressive EHR foundation models. The authors identify two problems with the conventional Global Stream (GS) approach: patients with longer trajectories receive disproportionate training exposure, and attention is allowed to cross patient boundaries, which is misaligned with downstream prediction based on a single patient's history. To address these issues, they propose a patient-level weighting controlled by an exponent α, together with Ordered Global Stream (OGS), which sorts patients by trajectory length and then samples windows according to their weights. OGS achieves the highest mean macro performance on four rollout-based tasks on MIMIC-IV v2.2.
Critical strengths
Clear problem definition. The paper clearly defines the distribution mismatch between pretraining and downstream application, and the problem is easy to follow.
Critical weaknesses
W1. Low methodological novelty and insufficient justification of the work's value
- Unclear distinction from NLP techniques: The way input sequences are composed is not itself new.
- The exponent-based reweighting of the form Mᵢ^α applies the same principle as temperature sampling in multilingual language models (Conneau et al., 2020) or data-mixture reweighting (Xie et al., 2023) at the document (patient) level.
- Similar techniques are readily found in NLP pretraining, including document-boundary masking (Zhao et al., 2024), grouping related documents into sequences (Shi et al., 2024), and packing that reduces truncation (Ding et al., 2024).
- The authors cite Xie et al., Zhao et al., Shi et al., and Ding et al. in their related work, but do not explain how the proposed methods differ from them. Moreover, the proposed methods rely only on a generic property, trajectory length, and do not appear to be designed specifically for the EHR domain.
Given the limited methodological novelty, the value of this paper rests on the generality and practical significance of its empirical findings. However, as discussed in W2, the evidence supporting these is insufficient.
W2. Insufficient evidence that the gains from OGS are generalizable and meaningful
- Evaluation limited to the zero-shot setting: All evaluations rely on rollout-based zero-shot prediction without fine-tuning. Under fine-tuning, a common way of using foundation models, the distribution mismatch between pretraining and downstream tasks that the authors target may be largely mitigated. In that case, the gains from OGS may have little practical significance. The authors should show whether their conclusions are limited to the zero-shot setting or also hold under fine-tuning or linear probing.
- Narrow range of downstream tasks: All four MIMIC-IV tasks are binary classification within a hospital or ICU stay, and their outcomes overlap, reducing to death (hospital/ICU mortality) and ICU admission (ICU admission/readmission). The original ETHOS paper (Renc et al., 2024), from which the evaluation protocol is adopted, also covered ICU length-of-stay estimation, first-day SOFA score regression, and DRG classification, none of which are included here. In addition, sepsis, hypertension, and kidney failure, which Figure 2c presents to show how outcome prevalence varies with trajectory length, are not evaluated. With this scope, it is difficult to judge whether OGS consistently improves performance.
- Small gains concentrated in threshold-dependent metrics: In the five-seed comparison (Table 5), the gains of OGS over GS+M on threshold-free metrics are within one standard deviation (AUROC 0.849 ± 0.008 vs. 0.845 ± 0.004; AUPRC 0.499 ± 0.007 vs. 0.492 ± 0.004). Gains exceeding one standard deviation appear only in metrics computed at a 0.5 threshold (F1 0.459 ± 0.017 vs. 0.430 ± 0.015; BAcc 0.679 ± 0.014 vs. 0.660 ± 0.009). The single-seed task-level results (Table 10) show that the F1 gains come from the mortality tasks, whose low prevalence makes F1 highly sensitive to a fixed threshold (hospital mortality 0.265 → 0.353; ICU mortality 0.431 → 0.497), while ICU admission and ICU readmission show almost no change. These gains may therefore reflect a change in the calibration of rollout risk rather than improved discrimination. This matters particularly in the medical domain, where the appropriate threshold depends on the clinical context (e.g., ICU early warning systems require a high-specificity operating point to reduce alarm fatigue). An evaluation that depends on a specific threshold does not sufficiently demonstrate the value of this work.
- Possible optimistic bias from single-seed results: On the test set of the same MIMIC-IV v2.2 setting with ETHOS tokenization, the single-seed results (Table 10) show gains of OGS over GS+M of +0.012 AUROC, +0.016 AUPRC, and +0.039 F1. In contrast, the five-seed means (Table 5) show gains of only +0.004, +0.007, and +0.029, respectively. Since all four generalization experiments are single-seed results, their gains may also be overestimated.
- Effects of sorting and weighting are not disentangled: OGS differs from GS+M in two respects at once: length-based sorting and window reweighting. Ablations that isolate each component are needed, for example, comparing a stream with weighting but no sorting against OGS with α=1, which approximately corresponds to sorting alone.
References
- Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 8440–8451, 2020.
- Ding, H., Wang, Z., Paolini, G., Kumar, V., Deoras, A., Roth, D., and Soatto, S. Fewer truncations improve language modeling. In Proceedings of the 41st International Conference on Machine Learning (ICML), pp. 11030–11048, 2024.
- Renc, P., Jia, Y., Samir, A. E., Was, J., Li, Q., Bates, D. W., and Sitek, A. Zero shot health trajectory prediction using transformer. npj Digital Medicine, 7(1):256, 2024.
- Shi, W., Min, S., Lomeli, M., Zhou, C., Li, M., Lin, X. V., Smith, N. A., Zettlemoyer, L., Yih, W., and Lewis, M. In-context pretraining: Language modeling beyond document boundaries. In International Conference on Learning Representations (ICLR), 2024.
- Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P., Le, Q. V., Ma, T., and Yu, A. W. DoReMi: Optimizing data mixtures speeds up language model pretraining. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
- Zhao, Y., Qu, Y., Staniszewski, K., Tworkowski, S., Liu, W., Miłoś, P., Wu, Y., and Minervini, P. Analysing the impact of sequence composition on language model pre-training. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 7897–7912, 2024.
Questions for authors
- Compared with the NLP techniques mentioned above (temperature sampling, data-mixture reweighting, and document packing), what is the methodological novelty of the proposed methods?
- Do the gains from OGS persist under fine-tuning or linear probing? Do they hold consistently on the tasks from the original ETHOS paper (ICU length of stay, SOFA score, and DRG classification) and on tasks related to the conditions in Figure 2c? Can the authors show that the improvements do not depend on a particular threshold?
Response to Reviewer DXkY
We thank the reviewer for their thorough review and constructive criticism, pointing out the connections to prior NLP work and asking where the zero-shot gains hold.
Relationship to NLP. We agree that our methods draw on temperature sampling, reweighting, boundary masking, and packing. Our contribution is to adapt and test these ideas for zero-shot patient trajectory prediction. In a global stream, patients with longer histories contribute more training tokens. Figure 2 shows that trajectory length is also associated with differences in clinical content and outcome prevalence. We use length to control patient exposure during pretraining and test how that affects predictions made directly from generated trajectories, without task-specific adaptation. We see this as a novel application of sequence composition in EHR modelling. Our goal is to shape the distribution of generated patient trajectories during pretraining itself, so the same training phase also aligns the model for zero-shot use, without the need for task alignment finetuning.
Evaluation scope. We study models intended for zero-shot use, where a single pretrained model can be evaluated on new clinical tasks specified at inference time without task-specific training. Our question is how pretraining sequence composition affects that capability. Fine-tuning and linear probing introduce task-specific supervision and answer a separate question about performance after adaptation. Because rollout evaluation is compute-intensive, we chose four established clinical tasks drawing on ETHOS. These tasks give an indication of performance across several clinical outcomes. We agree that future work should explore ICU length of stay, SOFA score, and DRG classification, however in this work we were compute-bounded and choose to prioritize the four common and general tasks.
Thresholds, seeds, and ablations. We agree that the AUROC and AUPRC gains over GS+Mask are modest. Both gains are statistically significant in the paired patient-level tests on the primary benchmark, conditional on the trained models (General Response 2). General Response 3 compares OGS with GS+Mask across thresholds. Balanced accuracy improves at every threshold shown, while F1 improves at most thresholds. General Response 2 extends the generalisation results to five training seeds, and General Response 1 reports the sorting and weighting ablations.