- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes two estimators to make risk prediction with autoregressive EHR foundation models simulation more efficient. Such models estimate risk by Monte Carlo (MC) sampling: the risk is the fraction of sampled future timelines in which the outcome token appears. The authors prove both SCOPE and REACH estimators are unbiased. They also show that REACH has lower variance than MC and SCOPE for the same number of sampled timelines
Critical strengths
-
The work has rigorous theoretical grounding. Each estimator comes with formal proofs of unbiasedness and variance properties. The derivations and equations are detailed, so a reader can follow every step. Some core results are also machine-verified in Lean 4, which is uncommon in this area.
-
The supplementary material contains full proofs, a notation table, worked examples and a counterexample. It also includes synthetic Markov-chain experiments, calibration and Brier-score analyses, and full result tables with confidence intervals.
Critical weaknesses
-
The citation style does not follow the ICLR-required format.
-
ETHOS, the core prior work, is never properly introduced. The introduction refers to “reproducing the pipeline of an existing paper (Appendix I)” without naming it. The problem statement also says: “In a standard workflow, electronic health records are represented as token sequences.” This is not accurate. EHR data are mostly stored and modeled as tables. Turning them into event-token sequences is specific to ETHOS-style generative models. Without an introduction to ETHOS, the narrative is cut off for a general reader: it is unclear why risk should be estimated by sampling future token sequences.
-
The main motivation is the computational cost of autoregressive inference. However, there are many established ways to cut this cost without changing the estimator. For example, KV caching, quantization, prefix sharing across the 100 timelines of the same patient etc. The 1,500+ GPU-hour figure is estimated from one hour of the original ETHOS-ARES code (Appendix I). The paper never reports the cost of its own SGLang pipeline, which already includes KV and prefix caching. So it is unclear how much of the motivating cost standard engineering already removes, and how much SCOPE/REACH save on top of it. Please discuss these techniques as complementary to the proposed estimators. Please also report GPU hours and per-patient latency on the same hardware for the original ETHOS implementation, an optimized MC baseline, and SCOPE/REACH.
-
Figure quality is low and the font sizes are too small to read.
-
Sampling settings are missing. Temperature and top-k/top-p are not reported. Unbiasedness requires sampling exactly from the model distribution whose probabilities SCOPE and REACH use, so these settings need to be stated.
-
The relation to classical estimators is missing, and there is no importance-sampling baseline. SCOPE is a cumulative-hazard (compensator) estimator and REACH is conditional Monte Carlo, both classical tools. Appendix Q analyses importance sampling but never runs it as a baseline.
Questions for authors
-
How does each of the 11 outcomes map to a token (or set of tokens)? Were computed label tokens injected into the training data, and are labels derived from tokens or from the raw structured data?
-
Under the Brier criterion, SCOPE needs more tokens than MC on 5 of 11 outcomes. How does this fit with recommending SCOPE as the default, and with “calibration preserved throughout”?
-
Which of the theoretical results are verified in Lean 4: is it only the three listed in Section 4.1, or also Appendix H and Appendix Q?
2 comments
Official Comment to reviewer oQmH (Critical Weakness 6b and Questions)
6b. Importance-sampling baseline
Since importance sampling is a general estimation framework rather than a uniquely specified estimator, implementing an importance-sampling baseline requires selecting a proposal distribution, and the choice of proposal can substantially affect both variance and computational efficiency. We agree that empirical comparisons with well designed importance-sampling proposals could be informative.
However, our theoretical analysis already establishes a stronger result than an empirical comparison with any single proposal from a broad, natural class of importance-sampling methods. Specifically, Appendix R proves that REACH has variance no greater than any importance-sampling estimator whose proposal preserves the conditional distribution of non-outcome tokens. This class includes arbitrary history-dependent re-scalings of the outcome-token probability, including constant re-scalings, the most natural potential baseline. Consequently, adding baselines from this class would not change the established per-sample variance ordering.
More general importance-sampling proposals that modify the distribution of non-outcome tokens are not covered by this guarantee and could offer different variance–computation tradeoffs. Such proposals would require additional design choices about how to steer generated clinical trajectories. Additionally, proposals optimized for individual rare outcomes generally sacrifice the simplicity of reusing an unmodified trajectory distribution across an arbitrary set of outcomes, which is a principal motivation for SCOPE.
Questions
- We have expanded Appendix B with an additional table, which describes the token corresponding to each outcome, the clinical information used to generate it, and the clinical threshold used to define it. For lab and vital outcomes, the extreme bin boundaries for the relevant category were set to the clinical threshold, so that a single token captures each outcome definition (noted in Appendix B, Table 2 caption). No separate label tokens were added: each outcome token is an ordinary token produced by the tokenization of the underlying EHR data, present in the training data like any other event, and labels are defined by the appearance of that same token in the held-out timeline.
- It is indeed true that SCOPE requires more tokens for 5 out of 11 outcomes in the Brier score analysis on MIMIC-IV, while on the local dataset SCOPE requires no more tokens than MC for any outcome. However, it is worth noting that these BSS differences at 100 samples are all less than 0.01, meaning there is a less than 1% relative difference in Brier score; this is the sense in which calibration is preserved. We have more prominently acknowledged this fact in the body. We now recommend that SCOPE be considered alongside Monte Carlo for all outcomes, rather than as a default replacement. Since both estimators are computed from the same pool of sampled timelines at negligible additional cost, this recommendation adds no inference cost. [Lines 403-405, 430-431]
- Only the results listed in Section 4.1 have been verified in Lean 4: the unbiasedness of SCOPE and REACH, that REACH is a Rao-Blackwellization of MC, and that REACH consequently reduces variance relative to MC. The results in Appendices H and Q (now I and R) are supported by written proofs but have not been formally verified. We have revised the abstract, which previously could be read as stating otherwise. [Lines 25-27]
Official Comment to Reviewer oQmH (Critical Weaknesses 1-6a)
(Edited 11/18/2026 to elaborate on point 3) We thank reviewer oQmH for their thorough and relevant comments. We respond to the critical critiques and questions in order below:
1 - Citation Style
We apologize for this oversight where references were being formatted from an incorrect template and we have corrected it. See our reply to reviewer 3HgR for further details.
2 - Introduction of generative prediction framework
We agree that generative inference (e.g. as in ETHOS) should be more strongly motivated in this paper. In addition to the changes described in our reply to reviewer 3HgR, we have modified the earlier language in the introduction to highlight the particularities and advantages of this prediction method and where they differ from standard supervised approaches. [Lines 43-46, 51-53]
3 - Engineering and Hardware
We agree that discussing how much of this cost was removed by our inference engineering would strengthen the paper. However, we would like to emphasize that all of our existing efficiency calculations measure how much SCOPE and REACH save on top of these engineering changes, as they are all calculated in terms of tokens per patient, a hardware and pipeline agnostic unit. In order to increase the clarity of our motivation, we have expanded this motivating quote to include the figure that, using our own SGLang inference pipeline, computing 100 timelines for the MIMIC-IV held-out set of over 96k patients required 495.8 A100 GPU hours, and have also added per patient latencies for both pipelines [Lines 69-73]. These figures cannot be directly compared to the ETHOS figure in Appendix J, since the experiments of that paper utilize a different model, tokenization scheme, cohort, and so on. We believe these figures strengthen the motivation of our work as it reveals that the cost remains significant even after substantial engineering.
To further address the hardware costs specific to the estimators (created by storing and updating the estimators, potential differences in timeline length caused by the outcome avoiding distribution REACH samples from, and others), we have also added a hardware cost analysis to the appendix, which measures the per-token cost of each method on an RTX 5090. The per-token overhead is negligible for SCOPE (+0.2%, 95% CI [−0.8, +1.3]) and small for REACH (mean of +2.4%, [+1.8, +3.0] across all outcomes, worst is mortality with +9.1% [6.3%, 11.8%]), far smaller than the median 2.5× to 3.4× token reductions reported in the paper. This is a valuable additional analysis that continues to support the existing conclusions [Appendix C]. Regarding the requested same-hardware comparison against the original ETHOS implementation, we did not include ETHOS, as its implementation is closely tied to their choice of model, making a fair same-model comparison impossible.
4 - Figure Quality
We greatly appreciate this feedback. We have made considerable efforts to improve the main figures in the body of the paper. We increased font sizes where possible, re-sized and re-arranged elements, and replaced all PNGs with PDFs to ensure quality remains high regardless of scaling. [Figure 1, Figure 2, Figure 3]
5 - Sampling Settings
We agree that using any form of top p/k or temperature based sampling would bias the resulting estimator. No top p/k sampling was performed and temperature was kept as 1.0 throughout. This fact is now noted in the methods. [Lines 367-368]
6 - Relationship to classical estimators and IS baseline
a. Relationship to classical estimators
We agree that the connection to classical statistical estimators is important and can be made more explicit. The cumulative hazard observation is greatly appreciated. An additional search uncovered a paper describing a "total hazard" estimator for Markov Chains that matches SCOPE much more closely than our previous reference. The related work section has been updated to use this new reference. [Lines 240-241]
We agree that REACH is an instance of conditional Monte Carlo (Rao–Blackwellization) constructed in an augmented outcome-avoiding probability space. This connection is already described in terms of Rao-Blackwellization in Section 3, and REACH is explicitly derived as a Rao–Blackwellized estimator in Section 2.2 [Lines 205-206] and Appendix Q. Nevertheless, we agree that explicitly naming CMC improves the positioning of our contributions, and we now name this relationship explicitly in the manuscript. [Lines 234-235]