acceptodds
Under review as a conference paper at ICLR 2027

HealthSAW: Scaling Health World Model with a Forecast-Driven Bottleneck

Abstract

Longitudinal patient healthcare records are widely considered a key source of clinical intelligence, but incorporating them into foundation models through next-token prediction has achieved limited success. This paper introduces HealthSAW, a Health State–Action World Model that represents patient healthcare records in a learned higher-order representation space, using local encoders to abstract tokens about a patient's health observations into health state vectors and treatment tokens into action vectors. These state–action representations enable a new pretraining paradigm in which the world model can autoregressively predict future events, including diagnoses, treatments, and responses to treatment. To make these vectors capture the patient's health rather than trivial token patterns, HealthSAW introduces a forecast-driven bottleneck: local decoders generate the tokens of the next state/action rather than reconstructing the current ones, and access prior patient history only through these higher-order state–action representations. Our experiments on about 19 million patients and 28 billion healthcare record tokens establish scaling laws for HealthSAW and demonstrate its superior performance across a wide range of disease-risk and treatment-outcome prediction tasks, including generalization to health systems unseen during pretraining. The world model architecture also enables new zero-shot patient simulation capabilities for predicting post-treatment laboratory outcomes, while being and faster than token-level models in pretraining and simulation, respectively. Further studies demonstrate the necessity of abstraction into the higher-order representation space, the benefits of this abstraction for context efficiency, especially with richer patient observations, and the generalization advantage of the forecast-driven bottleneck.

Primary area:
foundation or frontier models, including LLMs
Keywords:
world model, EHR foundation models, disease risk prediction, longitudinal patient modeling, clinical simulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WNWWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HealthSAW, a world model for structured EHR. Each day of patient is split into three embeddings: (1) time, (2) observation (diagnoses, labs, etc.), (3) action (medications, etc.). Local encoder compresses and extracts each embedding, a global causal transformer models the sequence, and local decoders generate the next block's raw tokens. The decoders access history only through the global output, and everything is trained end to end with next-block prediction. The model is pretrained on a large dataset, and the authors fit a repetition aware mixture scaling law up to 3B parameters.

Critical strengths
  • HealthSAW, EHR-FM, SMB, and Apollo share corpus, schedule, and 1B scale, so the gains can be attributed to the design rather than to data or compute. The raw-history ablation seems informative. It reaches lower pretraining loss but worse downstream performance (SS-EHR AUPRC 0.464 vs. 0.496). This supports the trivial-learning argument and is relevant beyond this paper, since pretraining loss is commonly used for model selection in EHR foundation models.

  • It is a practical, general architecture. Event-level abstraction gives consistent gains across in-domain, cross-site, and few-shot settings. Because context length scales with events rather than tokens, it also yields real efficiency gains, which grow with event density.

Critical weaknesses
  • I think the "unseen health system" claim is somewhat overstated. The largest gain comes from SS-EHR, which is from a different hospital but the same national healthcare system as CLAIM, the major pretraining source. The gains on the external US cohorts (PanCan and EHRSHOT) are much smaller. I'm not sure how well this would generalize to a completely different healthcare system or coding vocabulary. Also It would be helpful if reports about seeds or confidence intervals are presented in order to check the smaller indomain and external margins.

  • I'm not fully convinced by the treatment simulation results. The EHR-FM SFT baseline barely beats a constant median (32.58 vs. 34.15 WAPE), and a simple last-observation baseline is missing. Also, the rollouts keep the recorded treatment, so this is more like predicting future lab values after treatment rather than evaluating what would happen under different treatments. I think the treatment response framing is too strong (e.g., "responses to treatment" in the abstract and the therapy-selection motivation in section 4).

Questions for authors
  • Can you report results on a cohort from a different national system or coding vocabulary, or an ablation pretrained without CLAIM? This would separate architectural generalization from distribution overlap with CLAIM.
  • Since each event aggregates all records from a patient-day and observations are assumed to precede actions, how do you prevent post-treatment measurements from leaking into same-day treatment predictions? Also, how are tokens ordered within each observation/action block for autoregressive decoding?
4 comments
AuthorsOpenReview ↗

Response to Reviewer WNWW

Thank you for your thoughtful review and constructive suggestions. We address your concerns below.

1. SS-EHR Transfer and the Contribution of Pretraining

We have clarified the distinction between CLAIM and SS-EHR in §4.1. Although both datasets come from the same country, they represent different healthcare data systems. CLAIM is a national administrative claims database, whereas SS-EHR contains richer hospital records, with 15.55 observation tokens per event compared with 2.59 in CLAIM. SS-EHR also includes approximately five years of records beyond CLAIM's coverage (Table 6).

To examine whether HealthSAW's advantage on SS-EHR depends on external pretraining, we trained both HealthSAW and EHR-FM from scratch using the same SS-EHR labels and compared them with their pretrained counterparts as shown below and report SS-EHR macro AUPRC (%) (Appendix D.4, Table 15).

Task groupEHR-FM scratchHealthSAW scratchEHR-FM pretrainedHealthSAW pretrained
Cancer (7)29.4942.2630.3247.02
Cardiovascular (5)48.1755.2148.1663.01
Neurodegenerative (4)23.5827.1523.8637.45
Overall (16)33.8542.5334.2849.63

HealthSAW outperforms EHR-FM across all disease groups even without external pretraining. This shows that its architectural advantage on SS-EHR does not depend on overlap with CLAIM during pretraining. Furthermore, HealthSAW benefits substantially more from joint CLAIM and MS-EHR pretraining, demonstrating that it can effectively transfer learned information to SS-EHR.

We have also clarified the scope of our generalization claims: SS-EHR evaluates transfer across healthcare data systems within the same country, while PanCan and EHRSHOT provide additional evaluations on independent U.S. cohorts.

2. Confidence Intervals on MS-EHR

We added patient-level bootstrap confidence intervals for all three MS-EHR metrics (Appendix D.4, Table 14). The paired analysis confirms statistically significant overall improvements over both EHR-FM and SMB (JEPA) across all three metrics. For example, HealthSAW's overall AUPRC improvement over EHR-FM is 0.92 percentage points (95% CI: 0.44–1.37).

3. Comparison with the Last Observed Laboratory Value

As suggested, we added a baseline that predicts each patient's last observed laboratory value, using the training-set median when no eligible measurement is available. An eligible measurement was available for 98.75% of test samples.

MethodMacro WAPE (%)
Last observed laboratory value41.15
Training-set median34.15
Per-task EHR-FM SFT32.58
EHR-FM rollout30.74
HealthSAW rollout27.28

HealthSAW achieves the lowest prediction error among all evaluated methods, including the task-specific EHR-FM SFT baseline and EHR-FM simulation. It also outperforms the last-observation baseline on 20 of 22 tasks (Table 3 and Appendix F.3). These results show that HealthSAW's simulation provides useful predictive information beyond simple laboratory-value extrapolation, without requiring task-specific fine-tuning.

4. Forecasting After Recorded Treatment and What-if Analysis

We revised §4.3 to clarify that our treatment benchmark evaluates future laboratory outcomes following recorded treatments, rather than causal treatment effects. Conditioning on the recorded treatment allows us to compare simulated outcomes directly with observed follow-up measurements. This provides a quantitative evaluation of HealthSAW's ability to forecast patient trajectories under recorded care.

We also conducted an exploratory what-if analysis by removing treatment-related records and associated laboratory values from the input. Across 22 tasks, the resulting forecast distributions showed a median relative Wasserstein-1 distance of 5.85% from the original forecasts (Appendix F.3, Table 28). This shows that HealthSAW can generate different predictive distributions under modified clinical contexts, providing an initial demonstration of its what-if simulation capability.

AuthorsOpenReview ↗

Response to Reviewer WNWW

5. Treatment-Day Laboratory Values

All 22 treatment prediction tasks target laboratory measurements from subsequent events, with prediction windows starting at least three days after the treatment day (Appendix F.1, Table 24). Treatment-day laboratory measurements are never used as prediction targets, thereby preventing same-day post-treatment measurements from leaking into the prediction targets.

Since the input includes the full treatment day, some laboratory values may have been recorded after treatment. To examine their impact, we removed all numeric laboratory values from the treatment-day input while retaining the laboratory codes and treatment records. We kept the test cohort and model checkpoints unchanged (Appendix F.3, Table 26). This affected 11,728 of 13,921 test samples.

Rollout modelOriginal macro WAPE (%)WAPE without treatment-day lab values (%)
EHR-FM30.7430.46
HealthSAW27.2827.43

Removing treatment-day laboratory values has little effect on predictive performance, and HealthSAW maintains its advantage over EHR-FM. Together with the strictly future outcome windows, this indicates that HealthSAW's performance advantage is not driven by access to treatment-day laboratory values.

6. Token Ordering Within Observation and Action Blocks

For each patient-day, we place the observation block before the action block. Within each block, tokens follow a fixed order by clinical type: for example, vital signs and laboratory tests precede diagnoses, while medication records precede procedures. Laboratory codes and their corresponding values remain adjacent. The local decoders follow this fixed serialization order. It does not imply the actual temporal order of measurements and treatments within a day, which is not available in our records.

Thank you for taking the time to read and consider our response. We will carefully address these points in our revised manuscript. Please let us know if there is anything further we can do to improve the rating.

Thank you for the response and the additional experiments. Most of my concerns are addressed, so I will keep my positive rating.

Reviewer_4ZgoOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents HealthSAW, a hierarchical EHR model that compresses each recorded day into observation and treatment representations and trains these representations to predict subsequent information. Its main contribution is this architecture and forecast-driven training design, supported by large-scale experiments examining disease prediction, post-treatment laboratory prediction, scaling, and computational efficiency.

The results support useful predictive representations and computational benefits. However, several claims go beyond the evidence, particularly those concerning underlying patient health, the necessity of abstraction, and the superiority of the approach for patient simulation.

Critical strengths

HealthSAW improves average disease-risk prediction over EHR models trained under approximately matched conditions. Architectural ablations provide additional evidence that its design contributes to these improvements. I consider this meaningful evidence of improved representation learning.

The study includes substantial longitudinal datasets, adaptation to external sites, and a public few-shot benchmark. Faster training and generation are valuable, particularly when outcome estimation requires multiple simulated futures.

Critical weaknesses

The strongest disease-prediction results follow supervised fine-tuning, including local training at external sites. These demonstrate adaptation, but do not isolate the benefit of external pretraining compared with training the same architecture on target-site data alone. Similarly, predicting laboratory values after recorded treatment does not establish treatment effects. Without calibration or validation of variability across simulated futures, the evidence does not establish that HealthSAW captures patient health or uncertainty more faithfully. The abstract’s broad dismissal of next-token prediction and claim that abstraction is necessary are thus insufficiently supported.

EHR-FM is a reasonable architectural baseline, but its simulation implementation has a fixed context without rolling it forward, while HealthSAW has different history and generation limits. Incomplete trajectories remain in the disease-risk denominator, allowing resource constraints to influence accuracy. ETHOS or Curiosity are particularly relevant missing comparators: they also generates patient futures and estimates outcomes from simulations, while operating directly on event tokens rather than compressed observation-action summaries. Its omission weakens both the novelty discussion and the evidence that compression improves simulation while preserving predictive quality.

Questions for authors

How often does EHR-FM stop because it exhausts its token budget before reaching the prediction horizon, separately for each task, and how does this compare with HealthSAW? Why was a rolling context not used to allow EHR-FM to continue generating? Do the reported accuracy and speed advantages persist when both models can reach the requested horizon?

How much of the simulation advantage remains when both models receive the same observed history and sufficient generation budgets to reach the outcome horizon? Authors should report resource-limit termination rates and their effect on estimated probabilities.

How does the proposed zero-shot simulation approach differ from ETHOS, and why is this directly relevant precedent neither discussed nor evaluated?

Is there evidence that the generated probabilities are calibrated and that variability across trajectories represents outcome uncertainty? The reported point-prediction metrics do not answer this question. It would be very useful.

Do you have a comparison isolating the contribution of external pretraining from target-site supervised learning? In particular, how does the externally pretrained model compare with the same architecture trained from scratch using the same target-site data?

A more informative scaling experiment would increase pretraining scale on external datasets and evaluate a separate target dataset under three settings: no local adaptation, fine-tuning with controlled target-site label budgets, and training from scratch on those same target-site labels. This would establish whether external pretraining provides transferable value and how that value scales.

For simulation, comparison with ETHOS under matched clinical inputs and across trajectory budgets would directly test the proposed efficiency-accuracy trade-off. Calibration and distributional evaluation would help determine whether compression preserves uncertainty as well as central predictions.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 4Zgo

Thank you for your thoughtful review. We address your concerns below with additional experimental evidence and clarifications.

1. Evidence for State–Action Abstraction

We would like to emphasize that the benefits of state–action abstraction are supported by dedicated ablation studies in §4.4 and §4.5, rather than inferred solely from comparisons with token-level baselines.

First, we randomized the observation–action grouping while preserving the architecture, training objective, corpus, and schedule. HealthSAW achieves 2.8% and 23.0% higher AUPRC on MS-EHR and SS-EHR, respectively, demonstrating the importance of clinically meaningful state–action organization beyond generic block compression (Table 4).

Second, as more clinical information is retained within each event, HealthSAW's advantage over EHR-FM increases, reaching 45% higher AUPRC and nearly 3× higher representation throughput at full event content (Figure 5). This demonstrates that the benefits of abstraction become more pronounced with richer clinical observations.

Finally, our forecast-driven bottleneck ablations show that allowing decoders to access historical tokens directly reduces pretraining loss but degrades downstream performance. Reconstruction-based two-stage approaches also underperform HealthSAW (§4.5).

Together, these controlled experiments demonstrate that clinically structured state–action abstraction and the forecast-driven bottleneck are key contributors to HealthSAW's predictive and computational advantages. We have also refined the abstract's wording from "necessity" to "importance" to reflect this evidence more precisely.

2. EHR-FM Generation Budget and Inference Efficiency

In the original evaluation across 16 tasks, an average of 27.21% of EHR-FM trajectories reached the 3,072-token limit. All requested trajectories, including incomplete ones, were retained in the denominator when estimating disease risk (Appendix G.1).

HealthSAW generates patient trajectories at the event level, with generation controlled by the prediction horizon and a maximum number of future events. In contrast, EHR-FM generates token by token under a token budget.

To examine whether these differences explain the performance gap, we conducted additional experiments with matched observed patient histories. We increased EHR-FM's token budget from 3,072 to 4,096 and additionally evaluated EHR-FM with its rollouts truncated at the same maximum number of generated events as HealthSAW.

Given EHR-FM's substantial rollout cost, we selected three representative tasks: Any dementia (highest token-limit termination rate), heart failure (largest AUPRC gap between the models), and breast cancer (representative cancer task).

TaskHealthSAWEHR-FM (3,072)EHR-FM (4,096)EHR-FM (event-matched)
Heart failure0.25700.15240.14880.1423
Any dementia0.08930.04030.03840.0380
Breast cancer0.07480.06520.06300.0612

Increasing EHR-FM's token budget does not improve AUPRC on any of the three tasks. Instead, performance decreases slightly with longer rollouts. Regarding computational efficiency, under the original 16-task simulation protocol, HealthSAW reduces mean inference time from 5.700 to 1.781 seconds per patient, achieving a 3.20× speedup over EHR-FM (Appendix G.4). These results demonstrate that HealthSAW's predictive advantage cannot be explained simply by EHR-FM's original token limit: even with a larger generation budget, EHR-FM fails to close the performance gap. HealthSAW also achieves substantially greater inference efficiency.

AuthorsOpenReview ↗

Response to Reviewer 4Zgo

3. Comparison with ETHOS

ETHOS is an important precedent for zero-shot patient simulation. Both ETHOS and EHR-FM generate clinical timelines token by token using autoregressive Transformers. In contrast, HealthSAW models time, observation, and action blocks globally while generating tokens locally, enabling more efficient patient simulation.

We selected EHR-FM as our primary token-level baseline because it could be trained on the same corpus and schedule as HealthSAW, with a similar model size, enabling a controlled architectural comparison. EHR-FM also belongs to the same architectural family as ETHOS. We have added ETHOS and Curiosity to Related Work (§2) and discussed ETHOS rollout in Appendix G.1.

To provide a direct comparison, we also evaluated the released ETHOS model on three disease-risk prediction tasks. We converted MS-EHR test histories into ETHOS's input format and evaluated all three models on the same ETHOS-eligible test anchors, using 20 trajectories per anchor.

TaskEligible anchorsModelAUPRCPPV@99Sens@99
Heart failure58,586HealthSAW0.261954.07%16.47%
EHR-FM0.155236.34%7.99%
ETHOS0.081622.76%4.12%
Any dementia46,367HealthSAW0.094423.77%11.78%
EHR-FM0.040511.30%4.81%
ETHOS0.02582.52%0.98%
Breast cancer52,119HealthSAW0.061014.28%7.57%
EHR-FM0.062613.99%7.39%
ETHOS0.02172.96%1.39%

HealthSAW achieves higher AUPRC than ETHOS on all three tasks, and higher PPV and sensitivity at 99% specificity than both token-level models across all three tasks.

4. Forecast Distributions and Trajectory Variability

We clarified in §4.3 that the treatment benchmark evaluates laboratory values observed after recorded treatments, rather than causal treatment effects.

To address the questions about uncertainty, we conducted two complementary analyses: whether the generated predictive distributions reflect observed outcomes, and whether variability across simulated trajectories provides information about prediction error.

First, we evaluated prediction-interval coverage on 13,921 test samples across 22 tasks (Appendix F.3, Table 27). Each sample uses 20 generated trajectories. When fewer than five trajectories provide valid readouts, we supplement them with the training-set median bin until five distributions are available. We then average the distributions and measure whether the observed laboratory value falls within each predicted percentile interval.

Predicted percentile intervalHealthSAW observed coverageEHR-FM observed coverage
45th–55th23.32%18.54%
40th–60th30.33%24.24%
30th–70th46.38%35.26%
20th–80th60.74%46.94%
10th–90th76.11%60.72%

HealthSAW achieves consistently higher empirical coverage across all evaluated intervals, with higher coverage on 19 of the 22 tasks at the nominal 80% interval. Importantly, its wider prediction intervals also achieve coverage substantially closer to their nominal levels, providing evidence that HealthSAW better captures the distribution of observed laboratory outcomes, beyond point-prediction accuracy.

Second, we investigated whether variation across generated trajectories reflects prediction uncertainty. For each sample, we measured the standard deviation of trajectory-level predicted means and grouped samples into within-task quartiles. The analysis includes 10,602 samples with at least two valid trajectories from each model.

Trajectory spreadHealthSAW normalized errorEHR-FM normalized error
Lowest quartile9.9%10.8%
Second quartile12.6%13.6%
Third quartile15.1%14.4%
Highest quartile18.7%19.6%

For HealthSAW, prediction error increases consistently with trajectory spread, indicating that greater variation across simulated futures is associated with less accurate predictions. This relationship is also reflected in the median within-task Spearman correlations of 0.192 for HealthSAW and 0.136 for EHR-FM, across 21 tasks.

Together, these analyses provide evidence that HealthSAW's generated distributions capture meaningful information about observed outcomes and that variability across trajectories reflects prediction difficulty. They strengthen our simulation results by evaluating distributional properties and uncertainty-related information, rather than relying solely on point-prediction metrics.

AuthorsOpenReview ↗

Response to Reviewer 4Zgo

5. External Pretraining and Scaling on SS-EHR

To isolate the contribution of external pretraining from target-site supervised learning, we compared HealthSAW and EHR-FM trained from scratch using the same SS-EHR labels with their counterparts pretrained on CLAIM and MS-EHR (Appendix D.4, Table 15). The results report macro-averaged AUPRC across 16 tasks:

ModelFrom scratchCLAIM + MS-EHR pretrainingImprovement
EHR-FM33.85%34.28%+0.43 pp
HealthSAW42.53%49.63%+7.10 pp

These results separate two important sources of HealthSAW's improvement. The from-scratch comparison shows that HealthSAW's architectural advantage persists even without external pretraining. Meanwhile, the substantially larger gain from pretraining demonstrates that HealthSAW can more effectively leverage information transferred from external EHR datasets. Thus, its improvements are not simply attributable to target-site supervision or access to additional pretraining data alone.

We further evaluated how downstream performance changes with increasing pretraining scale on the same 16 SS-EHR tasks, using all available target-site labels. A pretraining token count of zero denotes training from scratch, and scores are averaged equally across tasks.

Model sizePretraining tokensAUPRCPPV@99Sens@99
0.27B040.17%42.84%39.31%
0.27B13.0B46.10%44.98%43.35%
0.27B26.0B47.35%45.08%43.98%
0.27B50.7B47.68%45.45%44.40%
1B103.9B49.63%45.80%45.54%

At a fixed model size of 0.27B, increasing the pretraining token budget consistently improves downstream performance, providing direct evidence of the value of additional pretraining data independently of model size. The larger 1B model achieves further improvements with an increased model and pretraining budget. These results extend the original scaling analysis (§4.1 and Appendix C) to cross-system transfer.

Together, these experiments directly address the contribution and scalability of external pretraining: HealthSAW provides stronger predictions even when trained solely on target-site labels, gains substantially more from external pretraining than EHR-FM, and continues to benefit as pretraining scale increases.

Reviewer_oWXGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HealthSAW proposes a state–action world model for EHRs instead of token-by-token autoregression. Each visit's tokens are partitioned by role: observations (diagnoses, labs, vitals, imaging, diagnostic procedures) and actions (medications, therapeutic procedures, surgeries, referrals). Local bidirectional encoders compress each block into a single vector; a global autoregressive Transformer predicts the next slot; local decoders generate tokens of the next block from the global output. The entire stack is trained end-to-end with next-block cross-entropy. It improve existing EHR FM baselines on both in-domain data and cross-center data.

Critical strengths
  • The forecast-driven bottleneck is a genuinely novel and well-isolated design, and the ablation is well-presented. The variant that gives the decoder direct access to raw history tokens achieves a lower next-block NLL yet underperforms HealthSAW on both MS-EHR and SS-EHR downstream. This cleanly refutes the default assumption that lower pretraining loss implies a better representation on EHR data, and operationalizes the trivial predictions, e.g. chronic diagnoses recorded at every visit, repetitive drug lists, boilerplate labs. Independent of whether HealthSAW itself becomes the dominant architecture, this result reframes how EHR-FM pretraining should be evaluated: validation loss cannot be the north star.
  • The scale and control of the apple-to-apple comparison are above the field's norm. EHR-FM, SMB-JEPA, Apollo-masked, and HealthSAW are all pretrained from scratch at ~1B parameters on the identical joint CLAIM+MS-EHR corpus with the same schedule and token budget, then adapted with identical SFT recipes. Most published EHR-FM claims rest on comparisons where the authors control their own method's pretraining but inherit baselines from different papers. Combined with the +45% cross-center AUPRC margin, this is one of the stronger pieces of evidence that the next-token paradigm has been genuinely underperforming — not just differently configured.
Critical weaknesses
  • The LLM baseline is deliberately weak. The paper's central narrative is "specialized EHR foundation models beat general-purpose frontier LLMs." But Haiku/Luna are not frontier reasoning models — they are optimized for throughput. A fair comparison would include Claude 4.5 Sonnet or Opus and GPT-5.6 (full) at minimum.

  • Reproducibility is restricted, and the numbers live where no one else can get. Four of five datasets (CLAIM, MS-EHR, SS-EHR, PanCan) are private, each behind a different DUA, with no plan to release model weights, representations, or even derived cohorts. The one public dataset is EHRSHOT, and that is precisely where HealthSAW's margin is more modest (+14–15% over the best tree, with EHR-FM already competitive). The dramatic +45% cross-center AUPRC is on SS-EHR, which no external group can audit. For a paper claiming a new paradigm for the field, this is a structural problem: readers must take the strongest result on faith. I recommend the authors run the full comparison on EHRSHOT and MIMIC-IV.

  • The cross-dataset result conflates "better generalization" with "richer events favor the obs/act decomposition." The paper itself shows HealthSAW's advantage grows monotonically with retained event content (Figure 5, 25%→100%), and SS-EHR has 55% more observation tokens per event than MS-EHR (15.55 vs 10.04, Table 6). This is interesting science, but it also means the +45% number is partly measuring an event-density effect, not pure cross-site generalization. The honest headline is "HealthSAW gains more when events are denser," not "HealthSAW generalizes 45% better across health systems." A fair cross-dataset test should match event density, e.g., by subsampling SS-EHR events to MS-EHR density, or by evaluating on MS-EHR-density external data.

Questions for authors
  • The cross-dataset +45% claim: can you report HealthSAW vs. EHR-FM on SS-EHR when SS-EHR events are subsampled to MS-EHR density, to disentangle event-richness benefit from cross-site generalization?

  • How does HealthSAW compare to BEHRT, Med-BERT, CEHR-BERT, or Foresight — i.e., existing visit-level EHR models that already pool within visits? Several of these predate SMB/JEPA and are the natural "visit-level" baselines rather than EHR-FM.

  • Will the authors release pretrained weights, patient representations on EHRSHOT, or at minimum the HealthSAW model trained on the public EHRSHOT subset, to enable community reproduction?

3 comments
AuthorsOpenReview ↗

Response to Reviewer oWXG

Thank you for your thoughtful review and constructive feedback. We address your concerns below with additional experiments and clarifications.

1. Comparison with Stronger LLMs

As suggested, we evaluated Claude Sonnet 5.5 and GPT-5.6 Sol on the same 4,000 test anchors per task as the EHR models (Appendix D.5, Table 16). Each cell reports macro AUPRC / Sens@99Spec / PPV@99Spec, averaged across 16 tasks.

ModelMS-EHR (%)SS-EHR (%)
Claude Sonnet 5.510.00 / 8.66 / 18.999.97 / 5.95 / 18.63
GPT-5.6 Sol10.78 / 8.83 / 14.5510.59 / 7.85 / 17.55
EHR-FM17.37 / 19.04 / 22.3333.60 / 31.44 / 38.47
HealthSAW19.01 / 22.15 / 24.5348.02 / 43.63 / 45.39

HealthSAW outperforms both stronger LLMs on all three metrics across both datasets. These results show that its predictive advantage extends beyond comparisons with the original Haiku and Luna baselines (Table 1)

2. Results on EHRSHOT and MIMIC-IV

To further validate HealthSAW on publicly accessible benchmarks, we evaluated it on both EHRSHOT and MIMIC-IV 3.1, covering different datasets and evaluation settings.

On EHRSHOT, we evaluated frozen pretrained representations across six new-diagnosis tasks. HealthSAW achieves higher macro AUPRC than EHR-FM and all published baselines at every evaluated label budget. At 128 examples per class, HealthSAW reaches 17.9% AUPRC, compared with 15.7% for EHR-FM and 15.6% for the strongest published tree baseline (§4.2, Appendix E.2). On MIMIC-IV 3.1, we evaluated 16 disease-risk prediction tasks, comparing HealthSAW with EHR foundation models and traditional machine learning baselines. HealthSAW also achieves the best performance across all four metrics on MIMIC-IV against the strongest competing baseline (Appendix E.3, Table 20).

ModelAUROC (%)AUPRC (%)Sens@99Spec (%)PPV@99Spec (%)
LightGBM75.019.407.5212.89
XGBoost72.1310.169.6013.65
Apollo66.648.887.3211.33
EHR-FM79.9212.4813.0717.39
SMB (JEPA)78.9513.3013.1617.30
HealthSAW-1B81.8013.4713.7317.65

Together, these results extend our evaluation beyond private datasets. EHRSHOT demonstrates the transferability of HealthSAW's frozen representations, while MIMIC-IV provides additional evidence of its predictive performance on an independently accessible clinical benchmark.

3. Release of Weights and Reproducibility

The private datasets used in our study are subject to data-use agreements that restrict their redistribution. To support independent reproduction and further research, we will release a HealthSAW model trained entirely on publicly accessible data, like EHRSHOT.

4. Effect of Event Density on SS-EHR Performance

We agree that event richness contributes to HealthSAW's advantage. Our event-content ablation (§4.4, Figure 5) shows that both models benefit from richer clinical events, with HealthSAW gaining more as event content increases.

To determine whether HealthSAW's advantage persists at lower event density, we conducted an additional experiment by retaining 64.57% of the clinical code–value units within SS-EHR events, reducing its average clinical tokens per event to approximately the MS-EHR level (9.02–9.03 versus 9.12–9.18). Both models were evaluated on the same 16 SS-EHR tasks.

MetricHealthSAWEHR-FMRelative gain
AUPRC25.79%22.59%+14.2%
PPV@9932.04%30.17%+6.2%
Sens@9924.04%21.03%+14.3%

HealthSAW retains its advantage across all three metrics. These results show that richer events amplify HealthSAW's advantage. Even when SS-EHR event density is reduced to approximately the MS-EHR level, HealthSAW still outperform EHR-FM. Thus, the cross-system performance gains are not solely attributable to differences in event richness.

AuthorsOpenReview ↗

Response to Reviewer oWXG

5. Comparison with BEHRT, Med-BERT, CEHR-BERT, and Foresight

We additionally evaluated all four models suggested by the reviewer on MS-EHR and SS-EHR. The results are summarized below, with each cell reporting macro AUPRC / Sens@99Spec / PPV@99Spec.

ModelMS-EHRSS-EHR
HealthSAW-1B0.1855 / 0.2285 / 0.24420.4974 / 0.4574 / 0.4583
BEHRT0.1599 / 0.2018 / 0.22460.4354 / 0.3992 / 0.4337
Med-BERT0.1500 / 0.1843 / 0.21190.4471 / 0.4175 / 0.4420
CEHR-BERT0.1616 / 0.2021 / 0.22500.4283 / 0.4000 / 0.4340
Foresight0.1694 / 0.2035 / 0.22810.4290 / 0.3975 / 0.4291

HealthSAW outperforms all four baselines on every reported metric across both datasets. We will also expand Related Work (§2) to discuss these models. An important architectural distinction is that incorporating visit information is not the same as performing longitudinal modeling at the event level. BEHRT, Med-BERT, and CEHR-BERT incorporate visit or temporal information into clinical-code sequences, while Foresight predicts clinical concepts autoregressively at the token level. In contrast, HealthSAW first compresses each observation and action block into a representation. Its global Transformer then models the sequence using only three positions per event—time, observation, and action—while local encoders and decoders handle individual clinical tokens (§3.1).

Therefore, HealthSAW differs from existing visit-aware models not merely in how it aggregates visit information, but in the basic units used for longitudinal sequence modeling. The additional comparisons demonstrate that this design provides consistent predictive benefits over these earlier approaches.

Thank you for taking the time to read and consider our response. We will carefully address these points in our revised manuscript. Please let us know if there is anything further we can do to improve the rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.