acceptodds
Under review as a conference paper at ICLR 2027

Sequence Composition in EHR Foundation Model Pretraining

Abstract

Autoregressive electronic health record (EHR) foundation models predict the next token in sequences of structured medical events. A common pretraining strategy concatenates patient histories into a global token stream and samples fixed-length windows from it. This gives patients with longer histories more training exposure and, under standard causal attention, allows tokens to attend across patient boundaries. These choices can misalign pretraining with downstream clinical prediction, where each prediction is based on a single patient’s history. We therefore investigate how the sampling and assembly of patient histories into pretraining sequences affect clinical prediction performance. We use the conventional global-stream approach as a baseline and compare alternative sequence composition methods that control how patient histories contribute to pretraining sequences. Across four MIMIC-IV v2.2 tasks, our best method improves macro AUROC, AUPRC, F1, and balanced accuracy, including relative gains of 9.3% in F1 and 2.7% in AUPRC under a fixed training recipe. We show that these gains extend across four additional dataset and tokenization settings, including INSPIRE and eICU-CRD, with higher macro AUROC, AUPRC, and F1 than the corresponding baselines in each case. Our findings show that restructuring existing EHR data can improve clinical prediction without changing the model or collecting additional data.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
electronic health records, foundation models, autoregressive pretraining, sequence composition, patient sampling, data weighting, zero-shot clinical prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DXkYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper analyzes how the way patient trajectories are sampled and assembled into pretraining sequences affects the zero-shot clinical prediction performance of autoregressive EHR foundation models. The authors identify two problems with the conventional Global Stream (GS) approach: patients with longer trajectories receive disproportionate training exposure, and attention is allowed to cross patient boundaries, which is misaligned with downstream prediction based on a single patient's history. To address these issues, they propose a patient-level weighting controlled by an exponent α, together with Ordered Global Stream (OGS), which sorts patients by trajectory length and then samples windows according to their weights. OGS achieves the highest mean macro performance on four rollout-based tasks on MIMIC-IV v2.2.

Critical strengths

Clear problem definition. The paper clearly defines the distribution mismatch between pretraining and downstream application, and the problem is easy to follow.

Critical weaknesses

W1. Low methodological novelty and insufficient justification of the work's value

  • Unclear distinction from NLP techniques: The way input sequences are composed is not itself new.
    • The exponent-based reweighting of the form Mᵢ^α applies the same principle as temperature sampling in multilingual language models (Conneau et al., 2020) or data-mixture reweighting (Xie et al., 2023) at the document (patient) level.
    • Similar techniques are readily found in NLP pretraining, including document-boundary masking (Zhao et al., 2024), grouping related documents into sequences (Shi et al., 2024), and packing that reduces truncation (Ding et al., 2024).
    • The authors cite Xie et al., Zhao et al., Shi et al., and Ding et al. in their related work, but do not explain how the proposed methods differ from them. Moreover, the proposed methods rely only on a generic property, trajectory length, and do not appear to be designed specifically for the EHR domain.

Given the limited methodological novelty, the value of this paper rests on the generality and practical significance of its empirical findings. However, as discussed in W2, the evidence supporting these is insufficient.


W2. Insufficient evidence that the gains from OGS are generalizable and meaningful

  • Evaluation limited to the zero-shot setting: All evaluations rely on rollout-based zero-shot prediction without fine-tuning. Under fine-tuning, a common way of using foundation models, the distribution mismatch between pretraining and downstream tasks that the authors target may be largely mitigated. In that case, the gains from OGS may have little practical significance. The authors should show whether their conclusions are limited to the zero-shot setting or also hold under fine-tuning or linear probing.
  • Narrow range of downstream tasks: All four MIMIC-IV tasks are binary classification within a hospital or ICU stay, and their outcomes overlap, reducing to death (hospital/ICU mortality) and ICU admission (ICU admission/readmission). The original ETHOS paper (Renc et al., 2024), from which the evaluation protocol is adopted, also covered ICU length-of-stay estimation, first-day SOFA score regression, and DRG classification, none of which are included here. In addition, sepsis, hypertension, and kidney failure, which Figure 2c presents to show how outcome prevalence varies with trajectory length, are not evaluated. With this scope, it is difficult to judge whether OGS consistently improves performance.
  • Small gains concentrated in threshold-dependent metrics: In the five-seed comparison (Table 5), the gains of OGS over GS+M on threshold-free metrics are within one standard deviation (AUROC 0.849 ± 0.008 vs. 0.845 ± 0.004; AUPRC 0.499 ± 0.007 vs. 0.492 ± 0.004). Gains exceeding one standard deviation appear only in metrics computed at a 0.5 threshold (F1 0.459 ± 0.017 vs. 0.430 ± 0.015; BAcc 0.679 ± 0.014 vs. 0.660 ± 0.009). The single-seed task-level results (Table 10) show that the F1 gains come from the mortality tasks, whose low prevalence makes F1 highly sensitive to a fixed threshold (hospital mortality 0.265 → 0.353; ICU mortality 0.431 → 0.497), while ICU admission and ICU readmission show almost no change. These gains may therefore reflect a change in the calibration of rollout risk rather than improved discrimination. This matters particularly in the medical domain, where the appropriate threshold depends on the clinical context (e.g., ICU early warning systems require a high-specificity operating point to reduce alarm fatigue). An evaluation that depends on a specific threshold does not sufficiently demonstrate the value of this work.
  • Possible optimistic bias from single-seed results: On the test set of the same MIMIC-IV v2.2 setting with ETHOS tokenization, the single-seed results (Table 10) show gains of OGS over GS+M of +0.012 AUROC, +0.016 AUPRC, and +0.039 F1. In contrast, the five-seed means (Table 5) show gains of only +0.004, +0.007, and +0.029, respectively. Since all four generalization experiments are single-seed results, their gains may also be overestimated.
  • Effects of sorting and weighting are not disentangled: OGS differs from GS+M in two respects at once: length-based sorting and window reweighting. Ablations that isolate each component are needed, for example, comparing a stream with weighting but no sorting against OGS with α=1, which approximately corresponds to sorting alone.

References

  • Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 8440–8451, 2020.
  • Ding, H., Wang, Z., Paolini, G., Kumar, V., Deoras, A., Roth, D., and Soatto, S. Fewer truncations improve language modeling. In Proceedings of the 41st International Conference on Machine Learning (ICML), pp. 11030–11048, 2024.
  • Renc, P., Jia, Y., Samir, A. E., Was, J., Li, Q., Bates, D. W., and Sitek, A. Zero shot health trajectory prediction using transformer. npj Digital Medicine, 7(1):256, 2024.
  • Shi, W., Min, S., Lomeli, M., Zhou, C., Li, M., Lin, X. V., Smith, N. A., Zettlemoyer, L., Yih, W., and Lewis, M. In-context pretraining: Language modeling beyond document boundaries. In International Conference on Learning Representations (ICLR), 2024.
  • Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P., Le, Q. V., Ma, T., and Yu, A. W. DoReMi: Optimizing data mixtures speeds up language model pretraining. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
  • Zhao, Y., Qu, Y., Staniszewski, K., Tworkowski, S., Liu, W., Miłoś, P., Wu, Y., and Minervini, P. Analysing the impact of sequence composition on language model pre-training. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 7897–7912, 2024.
Questions for authors
  1. Compared with the NLP techniques mentioned above (temperature sampling, data-mixture reweighting, and document packing), what is the methodological novelty of the proposed methods?
  2. Do the gains from OGS persist under fine-tuning or linear probing? Do they hold consistently on the tasks from the original ETHOS paper (ICU length of stay, SOFA score, and DRG classification) and on tasks related to the conditions in Figure 2c? Can the authors show that the improvements do not depend on a particular threshold?
1 comment
AuthorsOpenReview ↗

Response to Reviewer DXkY

We thank the reviewer for their thorough review and constructive criticism, pointing out the connections to prior NLP work and asking where the zero-shot gains hold.

Relationship to NLP. We agree that our methods draw on temperature sampling, reweighting, boundary masking, and packing. Our contribution is to adapt and test these ideas for zero-shot patient trajectory prediction. In a global stream, patients with longer histories contribute more training tokens. Figure 2 shows that trajectory length is also associated with differences in clinical content and outcome prevalence. We use length to control patient exposure during pretraining and test how that affects predictions made directly from generated trajectories, without task-specific adaptation. We see this as a novel application of sequence composition in EHR modelling. Our goal is to shape the distribution of generated patient trajectories during pretraining itself, so the same training phase also aligns the model for zero-shot use, without the need for task alignment finetuning.

Evaluation scope. We study models intended for zero-shot use, where a single pretrained model can be evaluated on new clinical tasks specified at inference time without task-specific training. Our question is how pretraining sequence composition affects that capability. Fine-tuning and linear probing introduce task-specific supervision and answer a separate question about performance after adaptation. Because rollout evaluation is compute-intensive, we chose four established clinical tasks drawing on ETHOS. These tasks give an indication of performance across several clinical outcomes. We agree that future work should explore ICU length of stay, SOFA score, and DRG classification, however in this work we were compute-bounded and choose to prioritize the four common and general tasks.

Thresholds, seeds, and ablations. We agree that the AUROC and AUPRC gains over GS+Mask are modest. Both gains are statistically significant in the paired patient-level tests on the primary benchmark, conditional on the trained models (General Response 2). General Response 3 compares OGS with GS+Mask across thresholds. Balanced accuracy improves at every threshold shown, while F1 improves at most thresholds. General Response 2 extends the generalisation results to five training seeds, and General Response 1 reports the sorting and weighting ablations.

Reviewer_D5M6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how patient trajectory composition during autoregressive electronic health record (EHR) foundation model pretraining affects downstream clinical prediction performance. The authors propose three sequence composition strategies, Patient Sampling (PS), Recursive Patient Sampling (RPS), and Ordered Global Stream (OGS), to control patient contributions using length-dependent sampling weights while maintaining patient boundaries. Experiments on MIMIC-IV, INSPIRE, and eICU-CRD show that OGS improves clinical prediction performance over conventional global-stream training, with relative improvements of 9.3% in F1 and 2.7% in AUPRC on MIMIC-IV

Critical strengths
  • interesting problem and problem formulation about ehr composition
Critical weaknesses
  • not a whole lot of novelty from a methods pov
  • The mechanism underlying the improvements remains unclear. further ablations can isolate contributions
  • potential confounding because trajectories are different lengths but training budget is fixed
  • not robust statistical testing
  • uses gpt 2 instead of exploring other sota or published ehr fms: motor, clmbr, context clues, ora
Questions for authors

address weaknesses in rebuttal

1 comment
AuthorsOpenReview ↗

Response to Reviewer D5M6

We thank the reviewer for raising points about the contribution, training budget, and choice of model.

Contribution and evidence. Our contribution is to test how patient exposure and sequence composition during pretraining affect zero-shot clinical prediction from generated patient trajectories. The sampling and masking methods draw on prior work in NLP. In EHRs, trajectory length is associated with differences in outcome prevalence and token composition. Both matter when the model is used to generate plausible future trajectories. We use sequence composition to shape which patient histories the model sees during pretraining, with the goal of better aligning it with the range of patients it will encounter at inference. To our knowledge, this is the first study to systematically test how patient-level pretraining sequence composition affects zero-shot rollout prediction in EHR models.

General Response 1 reports ablations of the sequence-composition choices, and General Response 2 reports the paired tests and five-seed generalisation results.

Fixed training budget. We hold the number of optimiser steps fixed at 500,000 and use the same batch size and context length across methods. As you note, methods can still see different amounts of usable data. Padding takes up space in each window without contributing prediction targets, so how efficiently a method uses each window is part of what we want to measure. Appendix Figure 9 shows that Patient Sampling is expected to cover less of the training corpus under the same window budget.

Model choice. We chose a familiar GPT-2-style backbone for this data-limited setting and kept it fixed to isolate the effect of pretraining sequence composition. Newer architectures may improve absolute performance. We expect the benefits of sequence composition to carry over to other backbones, and our controlled comparison tests those benefits on a fixed architecture. The published downstream results for MOTOR, CLMBR, and ORA use task-specific heads or linear probes. Context Clues reports zero-shot results in an appendix, but its main comparison uses task-specific linear probes trained on frozen representations. Those headline scores reflect supervised adaptation. Our comparison uses a shared zero-shot protocol to isolate the effect of pretraining sequence composition.

Reviewer_RRBBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how patient trajectories are assembled into training sequences for autoregressive EHR foundation models. The conventional Global Stream approach concatenates patient records, causing longer trajectories to contribute more training targets and permitting attention across unrelated patients.

The authors compare boundary-masked Global Stream with Patient Sampling, Recursive Patient Sampling, and Ordered Global Stream (OGS). OGS sorts trajectories by length, samples windows according to patient-contribution weights, and masks attention across patient boundaries. On four MIMIC-IV tasks over five seeds, OGS achieves the best mean macro metrics, including a 9.3% relative F1 improvement over Global Stream. Single-seed experiments on additional tokenizations, MIMIC-IV v3.1, INSPIRE, and eICU-CRD show directionally consistent improvements.

Critical strengths
  • The paper identifies an overlooked but consequential pretraining choice. Patient exposure and cross-patient attention can be changed without collecting data or modifying the model architecture. The GS+M comparison usefully separates attention leakage across patients from the effects of sequence composition.

  • The empirical study is broad. The primary experiment uses five training seeds, and the authors additionally test multiple datasets, institutions, and tokenization pipelines. The consistent direction of the external results suggests that the finding is not specific to one preprocessing pipeline.

Critical weaknesses
  • The causal source of OGS’s improvement is insufficiently isolated. OGS simultaneously changes patient weighting, sorts patients by trajectory length, changes window sampling, and uses a different candidate-window stride. Missing ablations—such as sorted GS+M without reweighting and weighted windows without sorting—make it unclear which component produces the gains. This limits the actionable scientific conclusion.

  • The claimed generalization does not show statistical significance. On the primary benchmark, AUROC and AUPRC improvements over GS+M are small relative to the reported variation, without paired significance tests. All four additional settings use only one training seed, despite several improvements being very small. Furthermore, each model receives only one stochastic rollout evaluation, and completion rates for rollouts excluded at the generation limit are not reported.

  • OGS bundles multiple interventions. OGS differs from GS+M in (a) sorting patients by length, (b) stride 64 vs 1, (c) weighted window sampling with α. Which component drives the gain? An ablation (sort-only with uniform window sampling; weighted-sampling without sort; sort+weight) would clarify.

Questions for authors

Can the authors provide ablations separating trajectory-length sorting, patient-contribution weighting, window stride, and boundary masking within OGS?

1 comment
AuthorsOpenReview ↗

Response to Reviewer RRBB

We thank the reviewer for identifying the need to separate the components of OGS and quantify uncertainty in the rollout results.

Ablations. In General Response 1, we test masking, sorting, and a change in stride individually on GS and compare weighted and unweighted GS at the same stride. Weighting improves all four metrics in that comparison, while changing the stride alone has little effect. Removing masking, sorting, or weighting from OGS also reduces performance across metrics. These ablations suggest that masking, sorting, and weighting each contribute to the gains of the full OGS configuration.

Statistical robustness. General Response 2 reports paired patient-level tests against GS+Mask on MIMIC-IV v2.2 with ETHOS tokenisation. The AUROC and AUPRC gains have p-values of 0.006 and 0.023, respectively, indicating statistical significance conditional on the trained models. We also extend the four additional generalisation settings to five training seeds. Most patient-level comparisons in these settings yield p-values below 0.05.

Rollout evaluation. General Response 3 reports trajectory completion rates for all methods in the primary experiment and for OGS and GS+Mask in the generalisation settings. We also repeat the evaluation at 20 rollouts per instance across five inference seeds on a fixed GS checkpoint. The amount of variation depends on the task and metric, with the largest variation in ICU mortality F1. OGS has the highest completion rate in the primary experiment and exceeds GS+Mask in all five generalisation settings.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response 1: Ablation Study

We thank the reviewers for highlighting the need for an additional analysis to determine which components of OrderedGlobalStream (OGS) drive the observed gains. We examine the contribution of each design choice below.

  1. Cross-patient masking
  2. Trajectory-length sorting
  3. Patient-contribution weighting
  4. Window stride of 64

Our original experiments already isolate the effect of cross-patient masking. To further investigate the impact of the other design differences, we ran an additional experiment in which we added each design choice to GS and removed each component from OGS one at a time where feasible. Due to compute limitations, we ran this experiment with a single training seed on the MIMIC-IV v2.2 dataset with ETHOS tokenization, using the same architecture and training recipes as in the original experiment (Section 4.3 of the manuscript). The limit on the number of elements that PyTorch’s WeightedRandomSampler can sample from, imposed by torch.multinomial, required us to use a stride of 64 for both OGS and GS with weighting. PyTorch issue #94131

Results

MethodModificationMacro AUROCMacro AUPRCMacro F1Macro BAcc
GSNone0.83770.47020.40200.6463
GS+ Mask0.84200.49050.42420.6559
GS+ Sorting0.83650.47090.40520.6501
GS+ Weighting (+ Stride = 64)0.85030.48390.43980.6646
GS+ Stride = 640.83920.47210.40150.6453
OGS− Mask0.85820.49500.45810.6832
OGS− Sorting0.85410.48980.44530.6539
OGS− Weighting0.84530.49210.43000.6533
OGSNone0.86110.50510.47880.6959

Discussion

On GS, masking and patient-contribution weighting improve performance across metrics, while sorting and changing the stride alone have little effect. The gains from weighting persist when we compare GS variants at the same stride.

Removing masking, sorting, or weighting from OGS reduces performance across all metrics, with weighting removal producing the largest F1 drop. This suggests that each component in collaboration contributes to the full OGS configuration.

AuthorsOpenReview ↗

General Response 2: Statistical Robustness

We thank the reviewers for highlighting the need for additional significance testing. We added a paired patient-level analysis for the primary MIMIC-IV v2.2 experiment and extended the generalization experiments to five training seeds.

Computational requirements

The high compute requirements of pretraining and autoregressive rollout evaluation limit how many experiments we can reasonably run. This is a common limitation in foundation model research (Bouthillier et al., 2021). In our experiments, each repetition requires pretraining a model from scratch, followed by computationally expensive rollout evaluation across multiple clinical prediction tasks. Each repetition requires approximately 48 H100-hours for pretraining and 42 H100-hours for evaluation.

We therefore consider the magnitude, direction, and consistency of improvements across metrics alongside formal statistical testing.

1. Statistical analysis of the primary experiments

Macro metricMean Δ (OGS − GS)95% CIWinsp-value
AUROC+0.0052[−0.0091, +0.0194]4/50.372
AUPRC+0.0132[−0.0029, +0.0291]5/50.086
F1+0.0387[−0.0029, +0.0802]4/50.061
BAcc+0.0244[−0.0045, +0.0533]4/50.079

OGS outperforms GS on at least four of the five seeds for each metric, and all four seed-level 95% confidence intervals include zero. To complement the seed-level analysis, we perform a paired patient-level randomization test. For each permutation, we randomly swap predictions between OGS and GS at the patient level and recompute the mean difference for each macro metric.

We perform 10,000 random permutations to estimate two-sided p-values and use 10,000 paired patient-level bootstrap resamples to construct 95% confidence intervals.

Macro metric95% bootstrap CIp-value
AUROC[0.0018, 0.0084]0.0043
AUPRC[0.0073, 0.0183]0.0001
F1[0.0312, 0.0456]0.0001
BAcc[0.0212, 0.0276]0.0001

In the patient-level analysis, all four p-values are below 0.05 and all four bootstrap confidence intervals exclude zero. This provides evidence for improvements over GS conditional on the trained models. The seed-level intervals above capture uncertainty across training runs and remain compatible with zero improvement.

2. Multi-seed generalization experiments

We now report the mean ± standard deviation of the paired differences (OGS − GS+Mask) across five training seeds for each dataset and tokenization setting. We include MIMIC-IV v2.2 with ETHOS tokenization as the reference setting alongside the four additional settings.

Dataset (Tokenization)AUROCAUPRCF1BAcc
MIMIC-IV v2.2 (ETHOS)+0.005 ± 0.009+0.007 ± 0.010+0.028 ± 0.022+0.020 ± 0.016
MIMIC-IV v2.2 (Ours)+0.024 ± 0.018+0.022 ± 0.012+0.038 ± 0.029+0.013 ± 0.012
MIMIC-IV v3.1 (ETHOS)+0.005 ± 0.010+0.005 ± 0.001+0.020 ± 0.020+0.012 ± 0.014
INSPIRE (Ours)+0.012 ± 0.015-0.004 ± 0.012+0.005 ± 0.021+0.003 ± 0.019
eICU-CRD (Ours)+0.003 ± 0.009+0.004 ± 0.007+0.013 ± 0.013+0.002 ± 0.005

OGS achieves positive mean improvements in 19 of the 20 comparisons, with gains in macro AUROC, F1, and balanced accuracy across all five settings.

We also applied the paired patient-level randomization test to OGS versus GS+Mask in all five settings and repor the calculated p-values for each metric and each setting in the table below. The first row addresses Reviewer RRBB's question about AUROC and AUPRC against the masked baseline on the primary MIMIC-IV v2.2 benchmark.

Dataset (Tokenization)AUROCAUPRCF1BAcc
MIMIC-IV v2.2 (ETHOS)0.00600.02300.00010.0001
MIMIC-IV v2.2 (Ours)0.00010.00010.00010.0001
MIMIC-IV v3.1 (ETHOS)0.00040.05110.00010.0001
INSPIRE (Ours)0.03140.43510.57010.3998
eICU-CRD (Ours)0.24890.05920.00010.0397

Fourteen of the 20 patient-level comparisons have p-values below 0.05.

Discussion

Across the tested settings, the five-seed mean gains are most consistent for AUROC, F1, and balanced accuracy. The patient-level tests provide complementary evidence conditional on the trained models, and the five-seed results show how much the gains vary across training runs.

AuthorsOpenReview ↗

General Response 3: Autoregressive Evaluation

We thank the reviewers for raising two questions about our zero-shot rollout evaluation. Reviewer RRBB asked how often rollouts reach the generation limit and how much results may vary with stochastic sampling. Reviewer DXkY asked whether the gains over GS+Mask depend on the 0.50 classification threshold, particularly for the low-prevalence mortality tasks. We report additional analyses addressing both concerns below.

1. Rollout completion rates

To address the concern about rollouts excluded at the generation limit, we report the percentage of attempted rollouts that reach a task-specific outcome before the limit.

ExperimentGSGS+MaskPSRPSOGS
Primary experiment · MIMIC-IV v2.2 (ETHOS)98.52%98.30%98.61%98.41%98.71%
Generalisation experiments · Average across five dataset and tokenisation settings—95.82%——96.68%

In the primary experiment, all methods complete at least 98.30% of attempted rollouts, with OGS having the highest rate at 98.71%. In the generalisation experiments, OGS has a higher completion rate than GS+Mask in all five settings, averaging 96.68% versus 95.82%. We exclude rollouts that reach the generation limit without a valid terminating token from the benchmark evaluation.

2. Sampling variability at 20 rollouts

To address Reviewer RRBB's concern about variability across stochastic evaluations, we held the GS checkpoint and test data fixed and measured performance across five inference seeds using 20 rollouts per prediction instance for each seed. The table reports mean ± standard deviation across seeds for each task.

Clinical taskAUROCAUPRCF1Balanced accuracy
Hospital mortality0.8548 ± 0.00670.2920 ± 0.00570.2583 ± 0.01130.5825 ± 0.0045
Hospital readmission0.7028 ± 0.00150.4822 ± 0.00220.3868 ± 0.00400.6095 ± 0.0020
ICU admission0.9131 ± 0.00130.7662 ± 0.00160.6924 ± 0.00230.7879 ± 0.0014
ICU mortality0.8841 ± 0.00520.4616 ± 0.01270.4308 ± 0.02140.6593 ± 0.0101
ICU readmission0.7415 ± 0.00640.4055 ± 0.00600.2871 ± 0.00820.5842 ± 0.0029

At 20 rollouts, the largest standard deviation was 0.0214 for ICU mortality F1, followed by 0.0127 for ICU mortality AUPRC and 0.0113 for hospital mortality F1. The other metric–task pairs had standard deviations of about 0.010 or less. Rollout sampling adds modest noise to most task–metric pairs, but it could affect comparisons when differences between methods are small. These results quantify inference variability and motivate the additional significance tests reported in General Response 2.

3. Classification threshold sensitivity

To address Reviewer DXkY’s concern that our gains might depend on the choice of threshold, we compared OGS with GS+Mask across operating points in the primary MIMIC-IV v2.2 experiment. We averaged macro F1 and balanced accuracy across paired seeds and computed differences before rounding.

ThresholdGS+Mask F1OGS F1Δ F1GS+Mask BAccOGS BAccΔ BAcc
0.100.39130.3808−0.01050.77070.7742+0.0036
0.200.47870.4702−0.00860.75450.7650+0.0104
0.300.49340.4971+0.00370.72210.7377+0.0156
0.400.47050.4868+0.01630.68870.7077+0.0190
0.500.43040.4587+0.02830.65970.6794+0.0197
0.600.38590.4120+0.02610.63480.6492+0.0144
0.700.31020.3433+0.03310.60190.6153+0.0134
0.800.26500.3018+0.03680.58540.5983+0.0129
0.900.19920.2255+0.02630.56240.5700+0.0076

OGS improves balanced accuracy at every threshold shown and F1 at most thresholds. GS+Mask has higher F1 at the lowest thresholds. At the original threshold, the F1 gains are concentrated in the mortality tasks, as the reviewer observed.

This analysis covers the zero-shot tasks in the manuscript. Calibration, fine-tuning, linear probing, and the additional outcomes the reviewer raised remain open questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.