- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes HealthSAW, a world model for structured EHR. Each day of patient is split into three embeddings: (1) time, (2) observation (diagnoses, labs, etc.), (3) action (medications, etc.). Local encoder compresses and extracts each embedding, a global causal transformer models the sequence, and local decoders generate the next block's raw tokens. The decoders access history only through the global output, and everything is trained end to end with next-block prediction. The model is pretrained on a large dataset, and the authors fit a repetition aware mixture scaling law up to 3B parameters.
Critical strengths
-
HealthSAW, EHR-FM, SMB, and Apollo share corpus, schedule, and 1B scale, so the gains can be attributed to the design rather than to data or compute. The raw-history ablation seems informative. It reaches lower pretraining loss but worse downstream performance (SS-EHR AUPRC 0.464 vs. 0.496). This supports the trivial-learning argument and is relevant beyond this paper, since pretraining loss is commonly used for model selection in EHR foundation models.
-
It is a practical, general architecture. Event-level abstraction gives consistent gains across in-domain, cross-site, and few-shot settings. Because context length scales with events rather than tokens, it also yields real efficiency gains, which grow with event density.
Critical weaknesses
-
I think the "unseen health system" claim is somewhat overstated. The largest gain comes from SS-EHR, which is from a different hospital but the same national healthcare system as CLAIM, the major pretraining source. The gains on the external US cohorts (PanCan and EHRSHOT) are much smaller. I'm not sure how well this would generalize to a completely different healthcare system or coding vocabulary. Also It would be helpful if reports about seeds or confidence intervals are presented in order to check the smaller indomain and external margins.
-
I'm not fully convinced by the treatment simulation results. The EHR-FM SFT baseline barely beats a constant median (32.58 vs. 34.15 WAPE), and a simple last-observation baseline is missing. Also, the rollouts keep the recorded treatment, so this is more like predicting future lab values after treatment rather than evaluating what would happen under different treatments. I think the treatment response framing is too strong (e.g., "responses to treatment" in the abstract and the therapy-selection motivation in section 4).
Questions for authors
- Can you report results on a cohort from a different national system or coding vocabulary, or an ablation pretrained without CLAIM? This would separate architectural generalization from distribution overlap with CLAIM.
- Since each event aggregates all records from a patient-day and observations are assumed to precede actions, how do you prevent post-treatment measurements from leaking into same-day treatment predictions? Also, how are tokens ordered within each observation/action block for autoregressive decoding?
4 comments
Response to Reviewer WNWW
5. Treatment-Day Laboratory Values
All 22 treatment prediction tasks target laboratory measurements from subsequent events, with prediction windows starting at least three days after the treatment day (Appendix F.1, Table 24). Treatment-day laboratory measurements are never used as prediction targets, thereby preventing same-day post-treatment measurements from leaking into the prediction targets.
Since the input includes the full treatment day, some laboratory values may have been recorded after treatment. To examine their impact, we removed all numeric laboratory values from the treatment-day input while retaining the laboratory codes and treatment records. We kept the test cohort and model checkpoints unchanged (Appendix F.3, Table 26). This affected 11,728 of 13,921 test samples.
| Rollout model | Original macro WAPE (%) | WAPE without treatment-day lab values (%) |
|---|---|---|
| EHR-FM | 30.74 | 30.46 |
| HealthSAW | 27.28 | 27.43 |
Removing treatment-day laboratory values has little effect on predictive performance, and HealthSAW maintains its advantage over EHR-FM. Together with the strictly future outcome windows, this indicates that HealthSAW's performance advantage is not driven by access to treatment-day laboratory values.
6. Token Ordering Within Observation and Action Blocks
For each patient-day, we place the observation block before the action block. Within each block, tokens follow a fixed order by clinical type: for example, vital signs and laboratory tests precede diagnoses, while medication records precede procedures. Laboratory codes and their corresponding values remain adjacent. The local decoders follow this fixed serialization order. It does not imply the actual temporal order of measurements and treatments within a day, which is not available in our records.
Thank you for taking the time to read and consider our response. We will carefully address these points in our revised manuscript. Please let us know if there is anything further we can do to improve the rating.
Thank you for the response and the additional experiments. Most of my concerns are addressed, so I will keep my positive rating.
Response to Reviewer WNWW
Thank you for your thoughtful review and constructive suggestions. We address your concerns below.
1. SS-EHR Transfer and the Contribution of Pretraining
We have clarified the distinction between CLAIM and SS-EHR in §4.1. Although both datasets come from the same country, they represent different healthcare data systems. CLAIM is a national administrative claims database, whereas SS-EHR contains richer hospital records, with 15.55 observation tokens per event compared with 2.59 in CLAIM. SS-EHR also includes approximately five years of records beyond CLAIM's coverage (Table 6).
To examine whether HealthSAW's advantage on SS-EHR depends on external pretraining, we trained both HealthSAW and EHR-FM from scratch using the same SS-EHR labels and compared them with their pretrained counterparts as shown below and report SS-EHR macro AUPRC (%) (Appendix D.4, Table 15).
HealthSAW outperforms EHR-FM across all disease groups even without external pretraining. This shows that its architectural advantage on SS-EHR does not depend on overlap with CLAIM during pretraining. Furthermore, HealthSAW benefits substantially more from joint CLAIM and MS-EHR pretraining, demonstrating that it can effectively transfer learned information to SS-EHR.
We have also clarified the scope of our generalization claims: SS-EHR evaluates transfer across healthcare data systems within the same country, while PanCan and EHRSHOT provide additional evaluations on independent U.S. cohorts.
2. Confidence Intervals on MS-EHR
We added patient-level bootstrap confidence intervals for all three MS-EHR metrics (Appendix D.4, Table 14). The paired analysis confirms statistically significant overall improvements over both EHR-FM and SMB (JEPA) across all three metrics. For example, HealthSAW's overall AUPRC improvement over EHR-FM is 0.92 percentage points (95% CI: 0.44–1.37).
3. Comparison with the Last Observed Laboratory Value
As suggested, we added a baseline that predicts each patient's last observed laboratory value, using the training-set median when no eligible measurement is available. An eligible measurement was available for 98.75% of test samples.
HealthSAW achieves the lowest prediction error among all evaluated methods, including the task-specific EHR-FM SFT baseline and EHR-FM simulation. It also outperforms the last-observation baseline on 20 of 22 tasks (Table 3 and Appendix F.3). These results show that HealthSAW's simulation provides useful predictive information beyond simple laboratory-value extrapolation, without requiring task-specific fine-tuning.
4. Forecasting After Recorded Treatment and What-if Analysis
We revised §4.3 to clarify that our treatment benchmark evaluates future laboratory outcomes following recorded treatments, rather than causal treatment effects. Conditioning on the recorded treatment allows us to compare simulated outcomes directly with observed follow-up measurements. This provides a quantitative evaluation of HealthSAW's ability to forecast patient trajectories under recorded care.
We also conducted an exploratory what-if analysis by removing treatment-related records and associated laboratory values from the input. Across 22 tasks, the resulting forecast distributions showed a median relative Wasserstein-1 distance of 5.85% from the original forecasts (Appendix F.3, Table 28). This shows that HealthSAW can generate different predictive distributions under modified clinical contexts, providing an initial demonstration of its what-if simulation capability.