acceptodds
Withdrawn from ICLR 2027

EveryQuery: A Promptable Foundation Model for Clinical Prediction Tasks over Electronic Health Records

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive foundation models for electronic health records can make predictions for diverse clinical tasks without finetuning. These methods work by generating possible “synthetic futures” for patients, and then inferring downstream predictions over these simulated trajectories. Despite their strong capabilities, these methods have several flaws. (1) Inference is extremely computationally expensive, as each prediction requires simulating many trajectories, each with many observations. (2) Their performance is noisy due to the variance inherent to simulation, which in particular reduces efficacy on rare events, which are often of high importance clinically. (3) They are not promptable, as their only input is the patient's medical history, requiring indirect aggregation at inference time. We introduce EveryQuery, a promptable foundation model for structured EHR data. Building on the insight that clinical prediction tasks can be written in a structured query language, EveryQuery takes a query as a prompt alongside the patient's history and estimates the outcome directly. This allows EveryQuery to make predictions across diverse clinical tasks, such as classification and survival modeling, without simulation or finetuning. Across three EHR datasets, EveryQuery outperforms a competitive autoregressive baseline on clinically relevant classification and time-to-event tasks, with mean AUROC gains of 14.3% and 12.7% on MIMIC-IV and a large academic medical center and comparable performance on the smaller NWICU dataset. Its advantage is largest for rare outcomes, and it requires less inference compute.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
clinical prediction, electronic health records, structured data, foundation model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_FEgBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper focuses on performing direct zero-shot predictions for clinical tasks with an EHR foundation model "EveryQuery", including binary classification and time-to-event predictions, instead of simulating trajectories that are expensive and inefficient, especially for rare events. Experiments demonstrated improved AUROC on three EHR datasets and improved inference efficiency compared with simulation of trajectories.

Critical strengths

The idea to prepend a query qq and perform p(y∣x,q)p(y | x, q) also conditioned on qq for zero-shot predictions is clearly described, and the experimental setup is also documented in detail.

Critical weaknesses

I have two main concerns about this paper evaluated at a conference level:

1. [Motivation and Correctness] This approach of prepending query for zero-shot inference is motivated under the assumption that "autoregressive EHR foundation models require simulated trajectories for downstream predictions". However, this is not necessarily true for existing autoregressive EHR foundation models, e.g., CLMBR-T-base is autoregressive and its transformer backbone can be freezed to only fine-tune a linear probe for few-shot prediction [1].

2. [Evaluation] The strength of the paper could be greatly improved by more comprehensive experiments. For the main results:

  • Section 5.1 & 5.3: The only baseline MEDS-EIC-AR [5] is an open-source model but not a peer reviewed paper (line 518 in reference). The paper chose it because "MEDS-EIC-AR is one of the few released AR foundation models that can be trained on any MEDS dataset" (cited). However, other strong peer-reviewed baselines such as CLMBR-T-base [1], Event Stream GPT [2], TransformEHR [3] are ignored, where they could be compared for a subset of tasks (e.g., binary classification) or evaluate both EveryQuery and these foundation models in a relaxed few-shot setting. Specifically, CLMBR-T-base natively supports MEDS datasets. It is not surprising that direct prediction is more efficient than trajectory simulation when the baseline is relatively weak.
  • Section 5.2: Larger-scale datasets such as EHRSHOT that cover more clinical tasks and rare events [4] will strengthen the results (e.g., what are the prevalence of the rare events evaluated in the paper). Additionally, the evaluation metric AUPRC (Area Under the Precision-Recall Curve) is more informative than AUROC in settings where positive labels are rare.
  • Section 5.3: In the evaluation, the autoregressive baseline MEDS-EIC-AR only generate K=20K=20 trajectories per patient. For rare events with prevalence less than 5% (6 out of 15 clinical tasks in EHRSHOT dataset [4]), the simulated trajectories will have very high variance of estimating the prevalence, so the results are also not as strong that EveryQuery improves from it.

References:

[1] EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models (NeurIPS 2023).

[2] Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Events (NeurIPS 2023).

[3] TransformEHR: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records (Nature Communications 2023).

[4] https://som-shahlab.github.io/ehrshot-website/tasks/.

[5] https://github.com/mmcdermott/MEDS_EIC_AR.

Questions for authors

While the initial results are promising, EveryQuery does not support out-of-distribution vocabulary for zero-shot predictions like many EHR foundation models, and the experimental results are not sufficiently strong considering its contribution to the literature.

Can the authors elaborate further on the novelty and motivation regarding when EveryQuery would be preferred over few-shot fine-tuning of a logistic regression head for inference?

Would it be possible to consider alternative peer-reviewed baselines and also include AUPRC for rare events as well as variance estimates as discussed in "Critical Weaknesses"?

1 comment
AuthorsOpenReview ↗

Response 1. Few shot evaluation

We thank the reviewer for their constructive feedback. We conducted additional experiments to address concerns about comparisons with other EHR foundation models, few-shot prediction, and the strength of our autoregressive baseline.

1. [Motivation and Correctness] "Autoregressive EHR foundation models [do not necessarily] require simulated trajectories for downstream predictions."

We agree that EHR foundation models can be adapted to downstream tasks through few-shot fine-tuning or linear probing. However, our paper focuses on zero-shot clinical prediction, where models answer previously unspecified tasks without task-specific labeled examples or training. To our knowledge, autoregressive generative EHR models are the only previously demonstrated approach to this setting. EveryQuery provides an alternative that directly predicts outcomes from structured queries without trajectory simulation.

a. "Other strong peer-reviewed baselines such as CLMBR-T-base [...] could be compared [...] in a relaxed few-shot setting."

We evaluated EveryQuery, MEDS-EIC-AR, and CLMBR-T-base in a few-shot representation learning experiment on MIMIC-IV tasks from our manuscript using the EHRSHOT few-shot evaluation protocol (Wornow et al., 2023). We adopted CLMBR's representation extraction and linear probing methodology and applied the same evaluation design to all three models to ensure a consistent comparison of representation quality. We trained L2-regularized logistic regression probes using K=4, 8, 16, 32, and 64 labeled examples per class. All probes were evaluated on the same held-out test set.

Table: Macro-averaged AUROC on 35 MIMIC-IV tasks at K=64 labeled examples per class.

ModelEvaluationAUROC ↑
EveryQueryZero-shot0.846
EveryQuery + linear probeFew-shot0.774
MEDS-EIC-AR + linear probeFew-shot0.752
CLMBR-T-base + linear probeFew-shot0.670

Performance improves with increasing labeled examples for all three models. EveryQuery representations outperform both baselines across all evaluated few-shot settings. Notably, EveryQuery's zero-shot performance exceeds its few-shot linear-probe performance by 0.072 AUROC. Since our few-shot evaluation uses a standardized CLMBR-style methodology rather than an approach optimized for EveryQuery, this gap suggests further opportunities to improve its few-shot adaptation.

Reviewer_XCeSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes EveryQuery, a query-conditioned foundation model for structured EHR data. Whereas existing autoregressive (AR) EHR foundation models obtain predictions by simulating future patient trajectories, EveryQuery represents a clinical task as a structured query composed of a code, a time window, and optional conditioning clauses, takes this query together with the patient history as input, and directly estimates the outcome probability. Experiments on three datasets (MIMIC-IV, NWICU, and a private AMC dataset) compare EveryQuery against MEDS-EIC-AR as the AR baseline, and the authors report superior performance on both classification and time-to-event tasks, an advantage that grows as events become rarer, and roughly 582× less inference compute.

Critical strengths

Well-defined problem. The cost and variance of simulation-based inference are real limitations of AR EHR models, and the idea of making the task itself an input to the model is simple and well motivated.

Critical weaknesses

W1. The entire evaluation rests on a single, apparently under-configured baseline

  • Missing EHR foundation models: All comparisons are made against MEDS-EIC-AR alone. Other EHR foundation models (Li et al., 2020; Rasmy et al., 2021; Steinberg et al., 2021; Kraljevic et al., 2024; Renc et al., 2024; Steinberg et al., 2024; Fallahpour et al., 2025; Kang et al., 2026) and task-specific supervised baselines are not evaluated.
  • Under-configured baseline: The baseline itself appears weak. It reaches chance-level AUROC on some one-day tasks (0.499 and 0.498 on MIMIC-IV in Table 7), which may result from inappropriate baseline settings, such as probabilities quantized by only K=20 trajectories. In addition, it is unclear whether the 768-token generation budget can reach horizons of up to 1,825 days.

This is not fixable by a minor amendment. Most of the experimental comparisons would need to be redesigned, with baselines from multiple model families under a common protocol.


W2. Concerns about task design and the reported statistical significance

  • Repeated values in Table 5: Table 5 contains many repeated values. For example, nine "Visit/IP within 365d | …, no RxNorm/X within 30d" tasks all report the same prevalence (0.222) and the same AUROCs (0.741 / 0.697). Such identical numbers raise the concern that some tasks may not be designed as intended. Moreover, results that effectively come from a single task may be counted as multiple wins in the aggregate metrics, which inflates the effective number of tasks underlying the win rates and significance tests.
  • Unreported uncertainty: The NWICU test set contains only 772 patients, and rare tasks likely have only tens of positive cases, yet no per-task confidence intervals or positive counts are reported. This matters most for the rare-event claim.

References

  • Fallahpour, A., Alinoori, M., Afkanpour, A., and Krishnan, A. EHRMamba: Towards generalizable and scalable foundation models for electronic health records. In Machine Learning for Health (ML4H), pp. 291–307, 2025.
  • Kang, B. G., Ahn, J., Han, H., and Yoon, S. Time-Conditioned Foreseeing: An EHR-specific foundation model for irregular dynamics and calendrical time. In Forty-third International Conference on Machine Learning (ICML), 2026. https://openreview.net/forum?id=IalpB5Mzaz
  • Kraljevic, Z., Bean, D., Shek, A., Bendayan, R., Hemingway, H., Yeung, J. A., Deng, A., Baston, A., Ross, J., Idowu, E., et al. Foresight—a generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling study. The Lancet Digital Health, 6(4), 2024.
  • Li, Y., Rao, S., Solares, J., Hassaine, A., Ramakrishnan, R., Canoy, D., Zhu, Y., Rahimi, K., and Salimi-Khorshidi, G. BEHRT: Transformer for electronic health records. Scientific Reports, 10(1):1–10, 2020.
  • Rasmy, L., Xiang, Y., Xie, Z., Tao, C., and Zhi, D. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. npj Digital Medicine, 4(1):86, 2021.
  • Renc, P., Jia, Y., Samir, A. E., Was, J., Li, Q., Bates, D. W., and Sitek, A. Zero shot health trajectory prediction using transformer. npj Digital Medicine, 7(1), 2024.
  • Steinberg, E., Jung, K., Fries, J. A., Corbin, C. K., Pfohl, S. R., and Shah, N. H. Language models are an effective representation learning technique for electronic health record data. Journal of Biomedical Informatics, 113, 2021.
  • Steinberg, E., Fries, J. A., Xu, Y., and Shah, N. MOTOR: A time-to-event foundation model for structured medical records. In International Conference on Learning Representations (ICLR), 2024.
Questions for authors
  1. Why were other EHR foundation models not included as baselines, and what explains the chance-level performance of MEDS-EIC-AR on the one-day tasks?
  2. Why do several tasks in Table 5 yield identical results, and how do the win rates and significance tests change after removing such duplicated tasks?
6 comments
AuthorsOpenReview ↗

Response 1. Missing EHR foundation models

We thank the reviewer for their detailed feedback. We conducted additional experiments to address concerns about the strength of our autoregressive baselines, inference efficiency, and evaluation methodology.

1. Missing EHR foundation models

a. Additional ETHOS baseline

We additionally compared EveryQuery against ETHOS (Renc et al., 2024), an independently developed autoregressive EHR foundation model capable of zero-shot prediction across different clinical tasks.

We initially planned to implement a dataset-agnostic version of ETHOS, but found that its preprocessing and tokenization involve numerous MIMIC-specific design choices that cannot be reasonably transferred to other datasets and vocabularies. Reproducing these choices in a dataset-agnostic manner would require substantial modifications, making it difficult to ensure a faithful comparison to the published baseline.

We therefore evaluated the original ETHOS implementation on MIMIC-IV, restricting the comparison to the 15 of our 38 tasks supported by ETHOS's vocabulary (e.g., ETHOS excludes MIMIC infusion codes).

Due to the high inference cost, we evaluated ETHOS with K=100 trajectories on 500 randomly sampled held-out patients. Even on this subset, ETHOS inference required 140 GPU-hours on an NVIDIA L40S.

Table 3: Comparison of EveryQuery, ETHOS, and MEDS-EIC-AR + SCOPE on the same 15 MIMIC-IV tasks and 500 randomly sampled held-out patients. All AR models use K=100 trajectories. Inference compute is reported in TFLOPs per patient.

MetricEveryQueryETHOSMEDS-EIC-AR with SCOPE
Macro-averaged AUROC ↑0.8350.7820.773
Inference TFLOPs / patient ↓0.0079130,00097.6
Relative inference cost ↓1×~16 million×~12,400×

We note that our MEDS-EIC-AR baseline with SCOPE achieves performance close to ETHOS (delta AUROC = -0.008, Wilcoxon signed-rank p=0.72), while requiring ~1,300× less inference compute. Even without SCOPE, the difference is not statistically significant (Wilcoxon signed-rank p=0.15). These results suggest that MEDS-EIC-AR is a competitive AR baseline, rather than an unusually weak implementation.

EveryQuery and ETHOS show no statistically significant difference in AUROC (Wilcoxon signed-rank p=0.98). Across all tasks, the median paired difference in AUROC is 0.015 and favors ETHOS. ETHOS wins on 9/15 tasks with relatively small win margins against EveryQuery. On the other hand, the macro-averaged AUROC favors EveryQuery. EveryQuery's gains on the remaining tasks are larger, resulting in a higher macro-averaged AUROC (0.835 vs. 0.782).

EveryQuery achieves comparable predictive performance to ETHOS while requiring ~16 million× less inference compute. Unlike ETHOS, EveryQuery also supports the complete dataset vocabulary.

b. Few-shot evaluations on EHR foundation models

We evaluated EveryQuery, MEDS-EIC-AR, and CLMBR-T-base in a few-shot representation learning experiment on MIMIC-IV tasks from our manuscript using the EHRSHOT few-shot evaluation protocol (Wornow et al., 2023). We adopted CLMBR's representation extraction and linear probing methodology and applied the same evaluation design to all three models to ensure a consistent comparison of representation quality. We trained L2-regularized logistic regression probes using K=4, 8, 16, 32, and 64 labeled examples per class. All probes were evaluated on the same held-out test set.

Table: Macro-averaged AUROC on 35 MIMIC-IV tasks at K=64 labeled examples per class.

ModelEvaluationAUROC ↑
EveryQueryZero-shot0.846
EveryQuery + linear probeFew-shot0.774
MEDS-EIC-AR + linear probeFew-shot0.752
CLMBR-T-base + linear probeFew-shot0.670

Performance improves with increasing labeled examples for all three models. EveryQuery representations outperform both baselines across all evaluated few-shot settings. Notably, EveryQuery's zero-shot performance exceeds its few-shot linear-probe performance by 0.072 AUROC. Since our few-shot evaluation uses a standardized CLMBR-style methodology rather than an approach optimized for EveryQuery, this gap suggests further opportunities to improve its few-shot adaptation.

AuthorsOpenReview ↗

Response 2. Baseline

2. Under-configured baseline

a. "inappropriate baseline settings, such as probabilities quantized by only K=20 trajectories"

We agree that K=20 Monte Carlo trajectories can limit the predictive performance of AR models, particularly for rare events. To address this concern, we evaluated up to K=100 trajectories and applied SCOPE variance reduction.

As detailed in our General Response: Compute–Accuracy Tradeoff between EQ and AR, stronger AR inference substantially improves performance. These experiments provide a more comprehensive comparison and clarify that EveryQuery's central advantage is its ability to make competitive predictions directly, without expensive trajectory simulation.

b. "baseline reaches chance-level AUROC on some one-day tasks"

We re-evaluated the two one-day procedure tasks cited by the reviewer using the same task definitions, including the conditions of no death and no timeline end within that day. The table reports mean AUROC across 100 paired patient resamples, with 95% percentile intervals (2.5th–97.5th percentiles). Each AR draw also subsamples K of 100 trajectories.

Setting224385 within 1d225792 within 1d
MC200.513 [0.469, 0.569]0.517 [0.485, 0.548]
MC1000.564 [0.507, 0.625]0.582 [0.540, 0.630]
SCOPE200.721 [0.666, 0.770]0.674 [0.639, 0.717]
SCOPE1000.750 [0.699, 0.800]0.698 [0.662, 0.733]
EveryQuery0.813 [0.773, 0.847]0.785 [0.755, 0.822]

With SCOPE, the strengthened AR baseline demonstrates above-chance discrimination on both tasks.

ETHOS cannot be evaluated on these tasks because its preprocessing excludes both target procedure codes.

c. "It is unclear whether the 768-token generation budget can reach horizons of up to 1,825 days."

We clarify that MEDS-EIC-AR used rolling generation with a maximum budget of 1,536 generated tokens, rather than the 768-token limit implied by its context window. We chose this budget to limit inference compute costs, which are already substantially higher for autoregressive models than for EveryQuery. Extending the generation budget is particularly expensive with rolling generation, as the KV cache must be recomputed when the context window rolls over, reducing the efficiency of optimized inference frameworks such as SGLang. Thus, increasing the generation budget would not only require generating more tokens, but also increase the computational cost per token.

We acknowledge that this budget limits the prediction horizon reached by some trajectories, particularly for longer-term tasks. Trajectories that do not reach the required horizon are discarded rather than treated as negative outcomes. We quantify the extent of this truncation below and will clarify the generation procedure and its limitations in the revised manuscript.

Table 1: Fraction of trajectories truncated before reaching the prediction horizon (K=100, MIMIC-IV).

Prediction horizonTrajectories truncated
6–12 hours~0–1%
24 hours~30%
48–72 hours~60%
7 days67.5%
28-day mortality70.7%
1-year mortality74.9%

These results highlight an additional limitation of simulation-based inference, because longer prediction horizons require substantially more generation compute, while truncation reduces the number of usable trajectories and may introduce selection bias.

d. "The baseline itself appears weak."

MEDS-EIC-AR was pretrained using the repository's default "medium" configuration. This translates to a 12-layer, decoder-only Llama transformer with a hidden dimension of 768, 12 attention heads, and approximately 102M non-embedding parameters.

The model was trained for 50,000 steps using AdamW, with a peak learning rate of 1 × 10⁻⁴ and weight decay of 0.01. The learning rate followed a cosine decay schedule after a linear warmup over the first 10% of training steps.

We will add these training details to the revised manuscript.

AuthorsOpenReview ↗

Response 3. Task design

3. Concerns about task design and the reported statistical significance

a. "Table 5 contains many repeated values. [...] Such identical numbers raise the concern that some tasks may not be designed as intended."

We agree that these nine tasks provide nearly redundant evidence and have removed all nine from the aggregate analysis. We also consolidated medication-dosage tasks and grouped related clinical questions into families before recomputing effect sizes, confidence intervals, and significance tests, as detailed below.

b. "Moreover, results that effectively come from a single task may be counted as multiple wins in the aggregate metrics, which inflates the effective number of tasks underlying the win rates and significance tests."

We agree that related tasks should not each count as independent evidence. We excluded all nine of "Visit/IP within 365d | …, no RxNorm/X within 30d" tasks with conditional clauses on SGLT2 medications. These tasks are nearly identical to the base hospitalization task and were removed from the revised evaluation.

On further review, we identified many medication tasks in AMC that differed only in the dosage of a particular RxNorm ingredient. For these, we retained one task per ingredient, choosing the highest prevalence task without reference to model performance. This removed 25 additional tasks.

Finally, we grouped tasks into clinical questions/families by merging related time horizons, lab test units, related diagnosis codes, and drug or procedure classes. Acute and long-term death, discharge, and visit outcomes form separate questions or families. The resulting analysis contains 19 families from 38 MIMIC-IV tasks, 13 from 27 NWICU tasks, and 30 from 66 retained AMC tasks. On AMC, we also pool tasks across different patient cohorts.

For each task and each of 100 draws, we compute the paired metric difference between EQ and the comparator using the same patient resample. Patient resamples are independent across tasks. The AR baseline MEDS-EIC-AR also subsamples K of 100 trajectories. We average differences within each family and give every family equal weight. EQ wins a family if its average difference across the 100 draws is positive. We compute 95% confidence intervals using 10,000 bootstrap samples. For each sample, we resample families with replacement, select one of the 100 draws, and average the selected families’ differences for that draw. We run Wilcoxon signed-rank tests on each family’s average difference across draws and apply Holm correction across 48 comparisons per metric: three datasets × two estimators × eight K values.

We compare the improved AR baseline with K=100 and SCOPE to EQ using this evaluation setting. On AUROC, EQ is significantly better on MIMIC-IV, significantly worse on AMC, and not significantly different on NWICU.

c. Investigating the effect of related conditioned clauses

We investigated whether future-conditioned clauses affect EveryQuery's performance. We removed all conditional clauses from the clinical tasks, retaining only the structured queries.

The patient cohorts differ between the two settings. Without conditional clauses, patients who do not satisfy the original conditions are no longer excluded.

EveryQuery's performance remains nearly identical. Spearman correlations exceed 0.99 across all datasets. The mean absolute AUROC difference is only 0.003 on MIMIC-IV and NWICU, and 0.013 on AMC. These results suggest that EveryQuery's performance does not depend on information provided by future-conditioned clauses.

Reviewer_rKFhOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EveryQuery which works as an alternative to autoregressive EHR models. Instead of predicting by simulating the patient trajectory, the model takes the patient history and question (in the form of a structured query) as input and predicts probabilities for yes/no. The model is trained by generating synthetic query combinations from the real dataset. The model outperforms an autoregressive baseline while achieving substantial efficiency improvements, as it does not need to generate long trajectories numerous times.

Critical strengths

The approach is interesting and well motivated. There are some interesting ideas in the architecture, such as training every query against all the possible codes. Achievement of the efficincy gains against autoregressive approaches is substantial and important. The structured query system is somewhat restrictive but still leaves some flexibility for complex queries. Evaluation is performed on 3 different datasets of different type and size.

Critical weaknesses

The autoregressive baseline is only able to generate 1024-256=768 tokens, yet the tasks include 7+ day horizons. Does the autoregressive model actually reach the end of the required horizon with this tight budget? How many trajectories get cut off and how is that counted in the evaluation?

One argued benefit of this model is that the query system allows to ask for predictions about any queries without requiring explicit training for those queries. But the model is trained by generating large amounts of training data from the same distribution as the test queries. Even the random-task evaluation is in-distribution for EveryQuery. It seems out-of-distribution generalisation of the queries is not demonstrated.

Most of the tasks have the form of "code c occurs within d days of prediction time | not censored, not dead". Given that, MOTOR is able to do the same tasks without autoregressive generation and should be included as a baseline. A simple multi-label multi-horizon head on the same backbone would also be a good comparison.

While the task is binary classification, only a ranking metric is reported. It would be good to know what the resulting precision and recall are, and how calibrated the resulting model is.

It would be good to better understand how the results achieved by both the AR baseline and the proposed EveryQuery model compare to previously published results. For example, ETHOS reports AUC 0.927 on the MIMIC ICU mortality task, whereas here the AR baseline achieves only 0.810. What is the reason for that? How do we know the baseline is sufficiently well trained?

Given that the time tokens are encoded explicitly in AR and fused in EveryQuest, does that mean the AR model sees less context? That seems unfair.

Training details of the AR baseline are not reported.

There are many decisions made in the model design and there are no ablation experiments to show their importance or benefit.


Update after author response:

It is good that you investigated the truncation percentages but these values look very problematic to me. Essentially more than half of the test data is thrown away in most settings. That means 1) that the performance is measured on datapoints sampled with a particular selection bias, and 2) the two systems are effectively not evaluated on the same datapoints.

I don't see why such a truncation was necessary. A rolling generation with a fixed context window could be run until the end token. It would take longer but it would at least give a prediction for every patient.

With this kind of truncation it seems the baseline results are not comparable, so I have to reduce my score unfortunately.

Given that most of the previously reported ETHOS scores on MIMIC-IV are around 0.8-0.95, it seems odd that the new macro-averaged result is only 0.78. A breakdown of these results would be needed to understand where the differences come from.

The MOTOR baseline was mentioned as a plan in the preliminary response but was not addressed in the final response.

Questions for authors

Please see above

3 comments
AuthorsOpenReview ↗

Response 1. Baseline

We thank the reviewer for their constructive feedback. We conducted additional experiments to address concerns about autoregressive inference, baseline performance, and evaluation metrics.

1. Underpowered autoregressive baseline

a. Stronger MEDS-EIC-AR baseline with SCOPE estimator

We agree that it is important to evaluate stronger AR inference configurations. We therefore extended MEDS-EIC-AR to K=100 trajectories and incorporated SCOPE variance reduction.

Please see our General Response: Compute–Accuracy Tradeoff between EQ and AR for the complete results across MIMIC-IV, NWICU, and AMC, including the substantial increase in inference compute required to strengthen AR predictive performance.

b. "The autoregressive baseline is only able to generate 1024-256=768 tokens, yet the tasks include 7+ day horizons."

We clarify that MEDS-EIC-AR used rolling generation with a maximum budget of 1,536 generated tokens, rather than the 768-token limit implied by its context window. We chose this budget to limit inference compute costs, which are already substantially higher for autoregressive models than for EveryQuery. Extending the generation budget is particularly expensive with rolling generation, as the KV cache must be recomputed when the context window rolls over, reducing the efficiency of optimized inference frameworks such as SGLang. Thus, increasing the generation budget would not only require generating more tokens, but also increase the computational cost per token.

c. "How many trajectories get cut off and how is that counted in the evaluation?"

We acknowledge that this budget limits the prediction horizon reached by some trajectories, particularly for longer-term tasks. Trajectories that do not reach the required horizon are discarded rather than treated as negative outcomes. We quantify the extent of this truncation below and will clarify the generation procedure and its limitations in the revised manuscript.

Table 1: Fraction of trajectories truncated before reaching the prediction horizon (K=100, MIMIC-IV).

Prediction horizonTrajectories truncated
6–12 hours~0–1%
24 hours~30%
48–72 hours~60%
7 days67.5%
28-day mortality70.7%
1-year mortality74.9%

These results highlight an additional limitation of simulation-based inference, because longer prediction horizons require substantially more generation compute, while truncation reduces the number of usable trajectories and may introduce selection bias.

AuthorsOpenReview ↗

Response 2. Precision Recall Metrics

2. "It would be good to know what the resulting precision and recall are, and how calibrated the resulting model is."

We evaluated AUPRC, Brier score, and expected calibration error (ECE) on the same clinical questions. We examine the effects of more trajectories, SCOPE at fixed K, and both changes. AUPRC differences are EQ minus AR; Brier and ECE differences are AR minus EQ, so positive values always favor EQ. The 95% confidence intervals are unadjusted; statistical significance is based on Holm-adjusted p-values.

AUPRC. Increasing trajectories and applying SCOPE improve AR AUPRC on all three datasets. EQ is significantly better on MIMIC-IV at MC20 and SCOPE20, but not at MC100 or SCOPE100. NWICU shows no significant difference in any settings. On AMC, there is no significant difference between the two except for an AR advantage at SCOPE100.

DatasetAR settingAUPRC difference95% CIHolm p
MIMIC-IVMC20+0.132
+0.084,+0.180+0.084, +0.180

| 0.004 | | MIMIC-IV | MC100 | +0.068 |

+0.020,+0.116+0.020, +0.116

| n.s. (0.319) | | MIMIC-IV | SCOPE20 | +0.104 |

+0.059,+0.148+0.059, +0.148

| 0.017 | | MIMIC-IV | SCOPE100 | +0.041 |

−0.002,+0.085−0.002, +0.085

| n.s. (0.681) | | NWICU | MC20 | −0.069 |

−0.151,+0.012−0.151, +0.012

| n.s. (0.879) | | NWICU | MC100 | −0.112 | [−0.192, −0.034] | n.s. (0.452) | | NWICU | SCOPE20 | −0.085 | [−0.168, −0.005] | n.s. (0.681) | | NWICU | SCOPE100 | −0.117 | [−0.190, −0.046] | n.s. (0.159) | | AMC | MC20 | +0.043 |

+0.012,+0.074+0.012, +0.074

| n.s. (0.305) | | AMC | MC100 | −0.041 | [−0.069, −0.013] | n.s. (0.105) | | AMC | SCOPE20 | −0.001 |

−0.028,+0.025−0.028, +0.025

| n.s. (1.000) | | AMC | SCOPE100 | −0.065 | [−0.090, −0.041] | 0.00004 |

Brier score. EQ has significantly lower Brier score on MIMIC-IV at MC20 and SCOPE20, but not at MC100 or SCOPE100. NWICU shows no significant difference. On AMC, AR has significantly lower Brier score at MC100 and SCOPE100, while MC20 and SCOPE20 are not significantly different from EQ.

DatasetAR settingBrier difference95% CIHolm p
MIMIC-IVMC20+0.026
+0.016,+0.038+0.016, +0.038

| 0.00076 | | MIMIC-IV | MC100 | +0.014 |

+0.003,+0.029+0.003, +0.029

| n.s. (0.060) | | MIMIC-IV | SCOPE20 | +0.022 |

+0.013,+0.034+0.013, +0.034

| 0.00076 | | MIMIC-IV | SCOPE100 | +0.009 |

+0.002,+0.020+0.002, +0.020

| n.s. (0.319) | | NWICU | MC20 | +0.000 |

−0.009,+0.015−0.009, +0.015

| n.s. (0.848) | | NWICU | MC100 | −0.004 |

−0.013,+0.010−0.013, +0.010

| n.s. (0.491) | | NWICU | SCOPE20 | −0.003 |

−0.010,+0.005−0.010, +0.005

| n.s. (0.517) | | NWICU | SCOPE100 | −0.007 | [−0.013, −0.001] | n.s. (0.408) | | AMC | MC20 | −0.007 |

−0.027,+0.004−0.027, +0.004

| n.s. (0.848) | | AMC | MC100 | −0.013 | [−0.034, −0.002] | 0.014 | | AMC | SCOPE20 | −0.009 |

−0.027,+0.002−0.027, +0.002

| n.s. (1.000) | | AMC | SCOPE100 | −0.014 | [−0.033, −0.003] | 0.008 |

Calibration (ECE). EQ has significantly lower ECE on MIMIC-IV at MC20, MC100, and SCOPE20, but not at SCOPE100. Neither NWICU nor AMC shows a significant ECE difference in any of these four settings.

DatasetAR settingECE difference95% CIHolm p
MIMIC-IVMC20+0.055
+0.039,+0.072+0.039, +0.072

| 0.00076 | | MIMIC-IV | MC100 | +0.024 |

+0.009,+0.047+0.009, +0.047

| 0.004 | | MIMIC-IV | SCOPE20 | +0.044 |

+0.030,+0.061+0.030, +0.061

| 0.00076 | | MIMIC-IV | SCOPE100 | +0.013 |

+0.003,+0.028+0.003, +0.028

| n.s. (0.271) | | NWICU | MC20 | +0.011 |

−0.007,+0.037−0.007, +0.037

| n.s. (1.000) | | NWICU | MC100 | +0.007 |

−0.012,+0.034−0.012, +0.034

| n.s. (1.000) | | NWICU | SCOPE20 | +0.003 |

−0.007,+0.016−0.007, +0.016

| n.s. (1.000) | | NWICU | SCOPE100 | −0.001 |

−0.012,+0.012−0.012, +0.012

| n.s. (1.000) | | AMC | MC20 | −0.010 |

−0.042,+0.009−0.042, +0.009

| n.s. (1.000) | | AMC | MC100 | −0.017 |

−0.050,+0.004−0.050, +0.004

| n.s. (1.000) | | AMC | SCOPE20 | −0.013 |

−0.043,+0.006−0.043, +0.006

| n.s. (1.000) | | AMC | SCOPE100 | −0.017 |

−0.050,+0.003−0.050, +0.003

| n.s. (1.000) |

These results further characterize the performance–compute tradeoff. At the largest AR budget, differences vary by dataset and metric, while EveryQuery requires far less inference compute.

AuthorsOpenReview ↗

Response 3. Additional baselines and clarifications

3. "It would be good to better understand how the results achieved by both the AR baseline and the proposed EveryQuery model compare to previously published results."

We additionally evaluated ETHOS (Renc et al., 2024) using its original implementation on MIMIC-IV. We initially planned to implement a dataset-agnostic version of ETHOS, but found that its preprocessing and tokenization involve numerous MIMIC-specific design choices that cannot be reasonably transferred to other datasets and vocabularies. Reproducing these choices in a dataset-agnostic manner would require substantial modifications, making it difficult to ensure a faithful comparison to the published baseline.

We therefore evaluated the original ETHOS implementation on MIMIC-IV, restricting the comparison to the 15 of our 38 tasks supported by ETHOS's vocabulary (e.g., ETHOS excludes MIMIC infusion codes).

Due to the high inference cost, we evaluated ETHOS with K=100 trajectories on 500 randomly sampled held-out patients. Even on this subset, ETHOS inference required 140 GPU-hours on an NVIDIA L40S.

Table 3: Comparison of EveryQuery, ETHOS, and MEDS-EIC-AR + SCOPE on the same 15 MIMIC-IV tasks and 500 randomly sampled held-out patients. All AR models use K=100 trajectories. Inference compute is reported in TFLOPs per patient.

MetricEveryQueryETHOSMEDS-EIC-AR with SCOPE
Macro-averaged AUROC ↑0.8350.7820.773
Inference TFLOPs / patient ↓0.0079130,00097.6
Relative inference cost ↓1×~16 million×~12,400×

We note that our MEDS-EIC-AR baseline with SCOPE achieves performance close to ETHOS (delta AUROC = -0.008, Wilcoxon signed-rank p=0.72), while requiring ~1,300× less inference compute. Even without SCOPE, the difference is not statistically significant (Wilcoxon signed-rank p=0.15). These results suggest that MEDS-EIC-AR is a competitive AR baseline, rather than an unusually weak implementation.

EveryQuery and ETHOS show no statistically significant difference in AUROC (Wilcoxon signed-rank p=0.98). Across all tasks, the median paired difference in AUROC is 0.015 and favors ETHOS. ETHOS wins on 9/15 tasks with relatively small win margins against EveryQuery. On the other hand, the macro-averaged AUROC favors EveryQuery. EveryQuery's gains on the remaining tasks are larger, resulting in a higher macro-averaged AUROC (0.835 vs. 0.782).

EveryQuery achieves comparable predictive performance to ETHOS while requiring ~16 million× less inference compute. Unlike ETHOS, EveryQuery also supports the complete dataset vocabulary.

4. "Given that the time tokens are encoded explicitly in AR and fused in EveryQuery, does that mean the AR model sees less context? That seems unfair."

Both EveryQuery and MEDS-EIC-AR receive exactly the same clinical events as input. Although MEDS-EIC-AR represents elapsed time using additional tokens, we ensure that these tokens do not reduce the number of clinical events available to the AR model.

In fact, this means EveryQuery sometimes uses fewer clinical events than its maximum context capacity would otherwise allow. Thus, differences in performance cannot be attributed to EveryQuery receiving more patient history.

5. "Training details of the AR baseline are not reported."

MEDS-EIC-AR was pretrained using the repository's default "medium" configuration. This translates to a 12-layer, decoder-only Llama transformer with a hidden dimension of 768, 12 attention heads, and approximately 102M non-embedding parameters.

The model was trained for 50,000 steps using AdamW, with a peak learning rate of 1 × 10⁻⁴ and weight decay of 0.01. The learning rate followed a cosine decay schedule after a linear warmup over the first 10% of training steps.

We will add these training details to the revised manuscript.

Reviewer_sB62OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EveryQuery, an EHR foundation model that directly predicts answers to structured clinical queries without task-specific fine-tuning or autoregressive trajectory simulation. Its query language specifies an event, prediction window, and optional conditional clauses. During pretraining, synthetic queries are sampled from this language, and the model jointly predicts the occurrence of every vocabulary code for each query window. Across MIMIC-IV, NWICU, and a private academic medical-center dataset, EveryQuery substantially outperforms one autoregressive baseline on classification and time-to-event tasks, particularly for rare outcomes. It also reportedly requires approximately 582× less inference compute. Additional analyses examine temporal and ontology consistency and performance on randomly sampled queries.

Critical strengths
  • The query-conditioned formulation is novel and compelling. Replacing expensive trajectory simulation with direct prediction over a structured task language is an elegant alternative to autoregressive EHR modeling. It permits many tasks and horizons to share one model while dramatically reducing inference cost.

  • The reported improvements are large and consistent on two substantial datasets. EveryQuery performs especially well on rare events, where Monte Carlo estimates from autoregressive models are intrinsically noisy. Evaluation across curated and randomly sampled tasks, three datasets, and survival outcomes provides encouraging breadth.

Critical weaknesses
  • The empirical comparison is too dependent on a single, potentially disadvantaged baseline. The autoregressive baseline uses only K=20 sampled trajectories, producing severely quantized estimates and predictably poor sensitivity for rare events. The paper does not provide a compute–accuracy curve as K increases, compare against variance-reduced AR estimators, or include strong discriminative/multitask models on the tasks they support. The training budget, optimization details, and variability of the AR baseline are also insufficiently documented. Consequently, it is unclear how much of the large gain comes from EveryQuery itself versus an underpowered inference procedure.

  • Uniform code sampling is likely the mechanism behind rare-event gains. §3.3 draws boundary events and clause codes "uniformly over all codes." Under natural event frequencies, rare codes would receive almost no training signal; under uniform sampling they get the same signal as common codes. AR, by contrast, is trained on the natural distribution. The paper doesn't disentangle "promptable prediction" from "implicit rare-event rebalancing." Suggested ablation: train EveryQuery with frequency-weighted query sampling (matching the AR distribution) and see whether rare-event gains persist. If they vanish, the finding is "rebalance your training distribution," which is a different claim.

  • Future-conditioned clauses and statistical dependence among tasks complicate the evaluation. Many tasks condition on facts such as “no death” or “no timeline end” during the future prediction window. Because clause answers are model inputs, these conditions may provide information unavailable at prospective prediction time. Furthermore, many evaluated tasks are nested or nearly duplicated across horizons and conditional clauses. Treating them as independent observations in Wilcoxon tests and confidence intervals can substantially overstate statistical significance. Only one trained model per method and dataset appears to be evaluated, so training-seed uncertainty is unknown.

Questions for authors

How does the AR baseline perform with larger K, variance-reduced estimators such as those discussed in the paper, or an equivalent inference-compute budget? A compute–accuracy frontier would better support both the performance and efficiency claims.

2 comments
AuthorsOpenReview ↗

Response 1. Baseline

We thank the reviewer for their constructive feedback. We conducted additional experiments to address concerns about the strength of our autoregressive (AR) baselines, the number of sampled trajectories, and inference efficiency.

1. The empirical comparison is too dependent on a single, potentially disadvantaged baseline

a. "The autoregressive baseline uses only K=20 sampled trajectories [...or] compare against variance-reduced AR estimators."

We agree that K=20 trajectories may be insufficient to characterize the full predictive capability of autoregressive models. We therefore substantially strengthened MEDS-EIC-AR by increasing the number of trajectories to K=100 and applying SCOPE, a variance-reduced inference estimator.

Please see our General Response: Compute–Accuracy Tradeoff between EQ and AR for the complete results, including revised AUROC comparisons and statistical significance across all three datasets.

On AUROC, at the largest evaluated AR budget with K=100 and SCOPE, EveryQuery remains significantly better on MIMIC-IV, MEDS-EIC-AR is significantly better on AMC, and NWICU shows no statistically significant difference.

Importantly, these experiments demonstrate that stronger AR inference improves predictive performance, but at substantially greater computational cost.

b. "The paper does not provide a compute–accuracy curve as K increases."

We have evaluated the compute–accuracy tradeoff across K ∈ {1, 5, 10, 20, 40, 60, 80, 100}, using both standard Monte Carlo and SCOPE estimation.

The complete results are presented in our general response, along with inference-compute comparisons. We will also include the compute–accuracy frontier figure in the revised manuscript.

c. Published MIMIC-IV baseline: ETHOS

"The empirical comparison is too dependent on a single, potentially disadvantaged baseline."

We additionally compared EveryQuery against ETHOS (Renc et al., 2024). ETHOS's preprocessing and tokenization exclude certain event types from its vocabulary (e.g., MIMIC infusion codes), preventing evaluation on many tasks. Of our original 38 tasks, only 15 are supported by ETHOS.

Due to the high inference cost, we evaluated ETHOS with K=100 trajectories on 500 randomly sampled held-out patients. Even on this subset, ETHOS inference required 140 GPU-hours on an NVIDIA L40S.

Table 3: Comparison of EveryQuery, ETHOS, and MEDS-EIC-AR + SCOPE on the same 15 MIMIC-IV tasks and 500 randomly sampled held-out patients. All AR models use K=100 trajectories. Inference compute is reported in TFLOPs per patient.

MetricEveryQueryETHOSMEDS-EIC-AR with SCOPE
Macro-averaged AUROC ↑0.8350.7820.773
Inference TFLOPs / patient ↓0.0079130,00097.6
Relative inference cost ↓1×~16 million×~12,400×

We note that our MEDS-EIC-AR baseline with SCOPE achieves performance close to ETHOS (delta AUROC = -0.008, Wilcoxon signed-rank p=0.72), while requiring ~1,300× less inference compute. Even without SCOPE, the difference is not statistically significant (Wilcoxon signed-rank p=0.15). These results suggest that MEDS-EIC-AR is a competitive AR baseline, rather than an unusually weak implementation.

EveryQuery and ETHOS show no statistically significant difference in AUROC (Wilcoxon signed-rank p=0.98). Across all tasks, the median paired difference in AUROC is 0.015 and favors ETHOS. ETHOS wins on 9/15 tasks with relatively small win margins against EveryQuery. On the other hand, the macro-averaged AUROC favors EveryQuery. EveryQuery's gains on the remaining tasks are larger, resulting in a higher macro-averaged AUROC (0.835 vs. 0.782).

EveryQuery achieves comparable predictive performance to ETHOS while requiring ~16 million× less inference compute. Unlike ETHOS, EveryQuery also supports the complete dataset vocabulary.

d. "The training budget, optimization details, and variability of the AR baseline are also insufficiently documented."

MEDS-EIC-AR was pretrained using the repository's default "medium" configuration. This translates to a 12-layer, decoder-only Llama transformer with a hidden dimension of 768, 12 attention heads, and approximately 102M non-embedding parameters.

The model was trained for 50,000 steps using AdamW, with a peak learning rate of 1 × 10⁻⁴ and weight decay of 0.01. The learning rate followed a cosine decay schedule after a linear warmup over the first 10% of training steps.

We will add these training details to the revised manuscript.

AuthorsOpenReview ↗

Response 2. Tasks

2. There is statistical dependence among tasks that complicate the evaluation.

a. Many tasks condition on facts such as “no death” or “no timeline end” during the future prediction window. Because clause answers are model inputs, these conditions may provide information unavailable at prospective prediction time.

We investigated whether future-conditioned clauses affect EveryQuery's performance. We removed all conditional clauses from the clinical tasks, retaining only the structured queries.

The patient cohorts differ between the two settings. Without conditional clauses, patients who do not satisfy the original conditions are no longer excluded.

EveryQuery's performance remains nearly identical. Spearman correlations exceed 0.99 across all datasets. The mean absolute AUROC difference is only 0.003 on MIMIC-IV and NWICU, and 0.013 on AMC. These results suggest that EveryQuery's performance does not depend on information provided by future-conditioned clauses.

b. Furthermore, many evaluated tasks are nested or nearly duplicated across horizons and conditional clauses. Treating them as independent observations in Wilcoxon tests and confidence intervals can substantially overstate statistical significance.

We agree that related tasks should not each count as independent evidence. We excluded all nine of "Visit/IP within 365d | …, no RxNorm/X within 30d" tasks with conditional clauses on SGLT2 medications. These tasks are nearly identical to the base hospitalization task and were removed from the revised evaluation.

On further review, we identified many medication tasks in AMC that differed only in the dosage of a particular RxNorm ingredient. For these, we retained one task per ingredient, choosing the highest prevalence task without reference to model performance. This removed 25 additional tasks.

Finally, we grouped tasks into clinical questions/families by merging related time horizons, lab test units, related diagnosis codes, and drug or procedure classes. Acute and long-term death, discharge, and visit outcomes form separate questions or families. The resulting analysis contains 19 families from 38 MIMIC-IV tasks, 13 from 27 NWICU tasks, and 30 from 66 retained AMC tasks. On AMC, we also pool tasks across different patient cohorts.

For each task and each of 100 draws, we compute the paired metric difference between EQ and the comparator using the same patient resample. Patient resamples are independent across tasks. The AR baseline MEDS-EIC-AR also subsamples K of 100 trajectories. We average differences within each family and give every family equal weight. EQ wins a family if its average difference across the 100 draws is positive. We compute 95% confidence intervals using 10,000 bootstrap samples. For each sample, we resample families with replacement, select one of the 100 draws, and average the selected families’ differences for that draw. We run Wilcoxon signed-rank tests on each family’s average difference across draws and apply Holm correction across 48 comparisons per metric: three datasets × two estimators × eight K values.

We are continuing to investigate the effect of uniform code sampling on rare-event performance and will report those results separately.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Compute–Accuracy Tradeoff between EQ and AR

We thank the reviewers for raising concerns about the strength of our autoregressive (AR) baseline, particularly the use of only K=20 sampled trajectories and the absence of variance-reduced inference. We agree that evaluating stronger AR inference methods is essential to understanding the relative advantages of EveryQuery.

We have substantially strengthened our AR baseline by increasing the number of sampled trajectories from K=20 to K=100 and incorporating SCOPE (Solo et al., 2026), a variance-reduced estimator.

While AR predictive performance improves considerably, these improvements come at an enormous inference cost, with the strongest AR configuration requiring approximately 12,400× more inference compute than EveryQuery. Our results therefore demonstrate an important compute–accuracy tradeoff.

EveryQuery does not necessarily outperform AR at arbitrarily large inference budgets, but it achieves competitive or superior predictive performance at orders of magnitude lower computational cost.

Across datasets, the plots show ΔAUROC (Comparator − EveryQuery) as the number of sampled trajectories (K) increases. Negative values favor EveryQuery. The estimates use the revised task set and statistical analysis adjusting for related tasks.

E = EveryQuery
M = MEDS-EIC-AR with Monte Carlo
S = MEDS-EIC-AR with SCOPE
uppercase = differs from EQ (Holm p<0.05)
lowercase = not significant


MIMIC-IV: AUROC relative to EveryQuery vs. MEDS-EIC-AR

+0.00 |E
-0.02 |
-0.04 |                                     S          S           S
-0.06 |              S          S
-0.08 |        S                                        M           M
-0.10 |     S                    M           M
-0.12 |               M
-0.14 |         M
-0.16 |      M
-0.18 |   S
-0.20 |
-0.22 |    M
      ++-------------+----------+-----------+----------+-----------+---
       0             20         40          60         80          100
                                K trajectories

NWICU: AUROC relative to EveryQuery vs. trajectories per patient

+0.02 |        s     s          sm          sm         sm          sm
+0.00 |E    s         m
-0.02 |         m
-0.04 |
-0.06 |   S
-0.08 |      M
-0.10 |
-0.12 |
-0.14 |
-0.16 |
-0.18 |    M
      ++-------------+----------+-----------+----------+-----------+---
       0             20         40          60         80          100
                                K trajectories

AMC: AUROC relative to EveryQuery vs. trajectories per patient

+0.02 |                         s           S          S           S
+0.00 |E       s     s
-0.02 |
-0.04 |     S                                                       m
-0.06 |                                                 m
-0.08 |                          M           M
-0.10 |
-0.12 |               M
-0.14 |   S     M
-0.16 |
-0.18 |      M
-0.20 |
-0.22 |    M
      ++-------------+----------+-----------+----------+-----------+---
       0             20         40          60         80          100
                                K trajectories

Conclusion.

At the largest evaluated AR budget (SCOPE100), AR achieves significantly higher AUROC than EveryQuery on AMC (AR − EQ ΔAUROC +0.028, Holm p=0.00017), while EveryQuery remains significantly better on MIMIC-IV (AR − EQ ΔAUROC −0.034, Holm p=0.021); the difference on NWICU is not statistically significant. We view this as the expected shape of a compute–accuracy tradeoff:

AR can keep buying accuracy with additional simulation, whereas EveryQuery makes competitive predictions in a single forward pass at roughly 12,400× lower inference cost.

Additional information. For reference, the corresponding AUROC differences (EQ − AR) are:

KMIMIC MCMIMIC SCOPENWICU MCNWICU SCOPEAMC MCAMC SCOPE
1+0.226+0.177+0.186+0.058+0.215+0.145
5+0.169+0.103+0.070+0.001 (n.s.)+0.170+0.035
10+0.146+0.083+0.030 (n.s.)−0.010 (n.s.)+0.149+0.010 (n.s.)
20+0.127+0.066+0.003 (n.s.)−0.017 (n.s.)+0.121−0.007 (n.s.)
40+0.108+0.052−0.012 (n.s.)−0.022 (n.s.)+0.089−0.019 (n.s.)
60+0.096+0.043−0.017 (n.s.)−0.024 (n.s.)+0.070−0.023
80+0.088+0.038−0.020 (n.s.)−0.025 (n.s.)+0.057 (n.s.)−0.026
100+0.082+0.034−0.021 (n.s.)−0.026 (n.s.)+0.047 (n.s.)−0.028

We report inference compute as a function of sampled trajectories on MIMIC-IV. Each cell reports inference TFLOPs per patient.

KMEDS-EIC-ARMEDS-EIC-AR + SCOPE
1—0.98
5—4.88
10—9.76
209.7519.52
4019.4939.04
6029.2458.57
8038.9978.09
10048.7497.61

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.