- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper focuses on performing direct zero-shot predictions for clinical tasks with an EHR foundation model "EveryQuery", including binary classification and time-to-event predictions, instead of simulating trajectories that are expensive and inefficient, especially for rare events. Experiments demonstrated improved AUROC on three EHR datasets and improved inference efficiency compared with simulation of trajectories.
Critical strengths
The idea to prepend a query and perform also conditioned on for zero-shot predictions is clearly described, and the experimental setup is also documented in detail.
Critical weaknesses
I have two main concerns about this paper evaluated at a conference level:
1. [Motivation and Correctness] This approach of prepending query for zero-shot inference is motivated under the assumption that "autoregressive EHR foundation models require simulated trajectories for downstream predictions". However, this is not necessarily true for existing autoregressive EHR foundation models, e.g., CLMBR-T-base is autoregressive and its transformer backbone can be freezed to only fine-tune a linear probe for few-shot prediction [1].
2. [Evaluation] The strength of the paper could be greatly improved by more comprehensive experiments. For the main results:
- Section 5.1 & 5.3: The only baseline MEDS-EIC-AR [5] is an open-source model but not a peer reviewed paper (line 518 in reference). The paper chose it because "MEDS-EIC-AR is one of the few released AR foundation models that can be trained on any MEDS dataset" (cited). However, other strong peer-reviewed baselines such as CLMBR-T-base [1], Event Stream GPT [2], TransformEHR [3] are ignored, where they could be compared for a subset of tasks (e.g., binary classification) or evaluate both EveryQuery and these foundation models in a relaxed few-shot setting. Specifically, CLMBR-T-base natively supports MEDS datasets. It is not surprising that direct prediction is more efficient than trajectory simulation when the baseline is relatively weak.
- Section 5.2: Larger-scale datasets such as EHRSHOT that cover more clinical tasks and rare events [4] will strengthen the results (e.g., what are the prevalence of the rare events evaluated in the paper). Additionally, the evaluation metric AUPRC (Area Under the Precision-Recall Curve) is more informative than AUROC in settings where positive labels are rare.
- Section 5.3: In the evaluation, the autoregressive baseline MEDS-EIC-AR only generate trajectories per patient. For rare events with prevalence less than 5% (6 out of 15 clinical tasks in EHRSHOT dataset [4]), the simulated trajectories will have very high variance of estimating the prevalence, so the results are also not as strong that EveryQuery improves from it.
References:
[1] EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models (NeurIPS 2023).
[2] Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Events (NeurIPS 2023).
[3] TransformEHR: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records (Nature Communications 2023).
Questions for authors
While the initial results are promising, EveryQuery does not support out-of-distribution vocabulary for zero-shot predictions like many EHR foundation models, and the experimental results are not sufficiently strong considering its contribution to the literature.
Can the authors elaborate further on the novelty and motivation regarding when EveryQuery would be preferred over few-shot fine-tuning of a logistic regression head for inference?
Would it be possible to consider alternative peer-reviewed baselines and also include AUPRC for rare events as well as variance estimates as discussed in "Critical Weaknesses"?
Response 1. Few shot evaluation
We thank the reviewer for their constructive feedback. We conducted additional experiments to address concerns about comparisons with other EHR foundation models, few-shot prediction, and the strength of our autoregressive baseline.
1. [Motivation and Correctness] "Autoregressive EHR foundation models [do not necessarily] require simulated trajectories for downstream predictions."
We agree that EHR foundation models can be adapted to downstream tasks through few-shot fine-tuning or linear probing. However, our paper focuses on zero-shot clinical prediction, where models answer previously unspecified tasks without task-specific labeled examples or training. To our knowledge, autoregressive generative EHR models are the only previously demonstrated approach to this setting. EveryQuery provides an alternative that directly predicts outcomes from structured queries without trajectory simulation.
a. "Other strong peer-reviewed baselines such as CLMBR-T-base [...] could be compared [...] in a relaxed few-shot setting."
We evaluated EveryQuery, MEDS-EIC-AR, and CLMBR-T-base in a few-shot representation learning experiment on MIMIC-IV tasks from our manuscript using the EHRSHOT few-shot evaluation protocol (Wornow et al., 2023). We adopted CLMBR's representation extraction and linear probing methodology and applied the same evaluation design to all three models to ensure a consistent comparison of representation quality. We trained L2-regularized logistic regression probes using K=4, 8, 16, 32, and 64 labeled examples per class. All probes were evaluated on the same held-out test set.
Table: Macro-averaged AUROC on 35 MIMIC-IV tasks at K=64 labeled examples per class.
Performance improves with increasing labeled examples for all three models. EveryQuery representations outperform both baselines across all evaluated few-shot settings. Notably, EveryQuery's zero-shot performance exceeds its few-shot linear-probe performance by 0.072 AUROC. Since our few-shot evaluation uses a standardized CLMBR-style methodology rather than an approach optimized for EveryQuery, this gap suggests further opportunities to improve its few-shot adaptation.