- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes an add-on training strategy that supplements existing longitudinal EHR survival models without changing their architecture or survival objective. It addresses a mismatch between training at one prediction time per patient and making repeated predictions during follow-up. Instead of supervising only the latest eligible prediction time, called one landmark training (1L), the approach supervises three selected times along each patient’s record, called three landmark training (3L). At each landmark, the model uses only the available history and predicts the remaining time until the event or censoring. Validation covers eight tasks across EHRSHOT and MIMIC-IV, six model families, and five development folds. The plain 3L results improve integrated Brier score in all 48 configurations and concordance in 46 of 48. Additional experiments examine training budgets, cohort size, record truncation, and prediction variability. The paper also evaluates successive predictions within individual patients, showing that better temporal agreement does not necessarily imply more informative risk trajectories.
Critical strengths
-
The main strength is that the proposed strategy can supplement existing survival models without requiring a new architecture or survival objective. Its benefits are demonstrated across two different EHR datasets and six model families, including models trained from scratch and a probe using frozen pretrained representations. This breadth supports a practical contribution: researchers can improve how they use existing patient histories while retaining their established modeling approach. The plain 3L results in Table 5 provide substantial evidence for this benefit.
-
The paper examines longitudinal prediction behavior beyond average discrimination and prediction error. It distinguishes agreement between successive predictions from changes in forecast hazards and within-patient risk direction. The finding that improved agreement can coexist with flatter hazards and weak separation between case and control risk slopes is particularly useful. These analyses reveal limitations that standard survival metrics would miss and make the paper informative beyond its central training recommendation.
Critical weaknesses
-
The primary results do not consistently identify which configuration is being evaluated. The MOTOR-Probe gains in Tab. 1 correspond to the exploratory "+r" configuration in Table 5, rather than plain three landmark training. For example, the AMI concordance gain is reported as 16.46 percentage points, whereas plain three landmark training improves it by 5.80 points. Appendix H states that the augmented configuration also changes batching, age conditioning, and checkpoint selection. Its gains therefore cannot be attributed to additional landmark supervision alone. Figure 12 and Table 13 also report different five-year slope AUCs for apparently identical MOTOR-Probe configurations. For pancreatic cancer, the respective baseline and three landmark values are 0.493/0.542 and 0.432/0.448. These discrepancies affect central results and require reconciliation. The broad benefit in Table 5 remains, but confidence in the reported effect sizes and trajectory conclusions is reduced.
-
The experiments change both the number and placement of supervised prediction times. The baseline always uses the latest eligible landmark, whereas three landmark training includes earlier portions of the trajectory. The results therefore establish an advantage over latest-only training, but do not show whether several landmarks outperform one representative landmark per patient. A fixed randomly selected landmark would be the most useful missing control. The budget and uncertainty claims also need qualification. Optimizer updates are explicitly matched for MOTOR-Probe, while matching nominal input tokens for other encoders does not consistently match total training computation. The reported configuration counts indicate positive mean differences, rather than independently established improvements. Because the development folds overlap and the test population is fixed, variation across folds does not capture uncertainty from sampling different test patients.
Questions for authors
-
Which configuration produced the MOTOR-Probe row in Tab. 1 and each primary figure? Please provide the primary comparison using plain three landmark training and reconcile Fig. 12 with Tab. 13, including any differences in models, patients, or metric aggregation.
-
How does three landmark training compare with one landmark per patient selected uniformly from the ten landmark pool and then held fixed, using the same optimizer budget? This experiment would distinguish the benefit of landmark placement from the benefit of supervising several histories from each patient. Including a representative encoder trained from scratch would also strengthen the budget comparison.
-
Are landmark pairs from the same patient included in pooled concordance calculations? Please provide confidence intervals for the principal performance differences using patient-level resampling. The unusually high and variable integrated Brier scores in several configurations also need explanations and comparison with a simple survival reference evaluated with the same observations.
-
What justifies interpreting increasing fixed-horizon risk among eventual cases as evidence of better temporal prediction? Remaining event-free and receiving new information can legitimately lower predicted risk, even for someone who later experiences the event. Similarly, informative new observations can justify disagreement with an earlier prediction transported forward in time. Please clarify the interpretation of these diagnostics and the rationale for encouraging increasing risk through the ranking objective.
3 comments
Addressing weaknesses and questions (part II)
W2. Budget and uncertainty claims.
We agree with all three points.
- (i) Optimizer updates are matched only for MOTOR-Probe, and matching input tokens does not match total training compute (Table 11). Section 4.2 now says so.
- (ii) Configuration counts reflect the sign of mean differences, not independently established improvements and we clarify this in Section 4.1.
- (iii) We added 95% patient-level bootstrap intervals for the 3L − 1L difference (Tables 17-18). Patients are resampled 1,000 times with all their landmarks, paired across models and folds, for GRU, Transformer and MOTOR-Probe on all eight tasks. The remaining three model families will be added to the final version.
Q3. Concordance, confidence intervals and high IBS values.
Same-patient pairs.
They are included in our C-index computations, but make up at most 0.12% of comparable pairs. Excluding them changes C-index by at most 0.04 pp and the 3L − 1L difference by at most 0.012 pp, with no configuration changing sign. We clarify this in Appendix C.1.
Confidence intervals.
On the ten-landmark pool, the interval excludes zero in favor of 3L in 21/24 configurations for IBS and 16/24 for C-index, and never in favor of 1L (Tables 17-18). On the latest-landmark pool, the IBS gain remains clear (17/24), but the C-index gain is mostly within sampling uncertainty (8/24 in favor of 3L, 1/24 in favor of 1L). These intervals capture test-patient sampling given the trained models, while the fold standard deviations capture training variability. We report both.
Configurations whose 95% patient-level bootstrap CI of the 3L − 1L difference excludes zero. GRU, Transformer and MOTOR-Probe on eight tasks (24 configurations per pool). Full intervals in Table 17.
| Metric | Ten-landmark pool | Latest-landmark pool |
|---|---|---|
| C-index | 16/24 favor 3L · 0 favor 1L | 8/24 favor 3L · 1 favors 1L* |
| IBS | 21/24 favor 3L · 0 favor 1L | 17/24 favor 3L · 0 favor 1L |
* Transformer on AMI (−4.04 pp).
High IBS.
We agree that some IBS values remain unusually high, for instance for all models on HPT and for DeepHit/Dynamic-DeepHit on AMI (EHRSHOT) under the ten-landmark evaluation. A sanity check showed that such configurations do not beat a Kaplan-Meier reference (fitted on 3L training rows) under 1L or 3L, although 3L moves them closer. This points to poorly calibrated absolute risk, most severely on EHRSHOT HPT, which we will investigate further. The 3L − 1L IBS gains there therefore reflect reduced miscalibration, not well-calibrated absolute risk. We state this in Section 4.1. C-index results and the paired 3L − 1L contrast are unaffected.
Q4. Interpreting rising risk and TV.
We agree. Remaining event-free and new information can legitimately lower a case’s predicted risk and can justify disagreement with a transported prediction. We therefore treat neither rising case risk nor low TV as a target. Both are descriptive diagnostics, compared between 1L and 3L on identical patients and pairs. SlopeAUC is more informative because it contrasts cases with controls, who are subject to the same effects (Section 3.3, Appendix C.2). The ordering loss does assume that a case’s short-term risk rises before the event. We now state this assumption explicitly (Appendix H) and present the loss only as a proof of principle that trajectory direction can be shaped. In our experiments, it coincides with better case-control separation (Table 2). The assumption is task-dependent: recurrent or relapsing conditions call for learning risk dynamics directly rather than imposing monotonic increase. This discussion has been added to Section 5.
Follow-up
Dear Reviewer qL2W, we hope you don’t mind a brief follow-up before the discussion period ends. Following your review, we corrected the reporting errors and re-verified all tables, added the random-landmark control you suggested ($1L_{rand}$), and reported patient-level bootstrap intervals. We would value your feedback on whether these changes resolve your concerns, and we would be happy to answer any remaining question while there is still time. If you find our answers satisfactory, we would be grateful if you would consider updating your assessment. Thank you again for the care you put into this review, and for spotting the discrepancies in our reported results.
Addressing results discrepancies, weaknesses and questions (part I)
Thank you for your very careful review. You were right about the reporting discrepancies, and we sincerely apologize for them. We are grateful that you caught them, as correcting them made the paper more reliable. They came from tagging errors in our table-building scripts and are detailed under “Correction of reported results” in the general comment. For readability, all updated and additional elements are highlighted in blue in the currently revised manuscript.
W1 / Q1. Which configuration are reported, and former results mismatch.
i) On main Table 1 values.
The MOTOR-Probe row reported 3L + r. It now reports plain 3L (e.g., AMI C-index +5.87 pp). No other row was affected, and all primary figures already use plain 3L. The MOTOR-Probe gain remains, but is reduced compared to MOTOR-Probe + r. The updated overall gains have been corrected in Abstract and Section 4.1 (e.g., "median gains of 3.54 and 3.48 percentage points (pp)" instead of "4.34 and 3.48 pp" previously).
ii) On MOTOR-Probe 3L values.
Tables 2, 6 and 16, and Figure 6 used the update-matched run. They now use standard 3L training, while the update-matched variant appears only in Table 12. The update-matched run was a slightly more conservative version of the probe, but differences with the updated values are small (e.g., AMI C-index 63.30 vs. 63.38). This error did not overturn any claim.
iii) On SlopeAUC mismatch.
The EHRSHOT slopes in the former Table 13 (now Table 16) were computed on a position-stratified test pool instead of the time-stratified pool used elsewhere for this dataset. After recomputation, Table 2, Figure 6 and Table 16 agree (pancreatic cancer: 0.493 / 0.539 / 0.611 for 1L / 3L / 3L + r).
iv) Regarding the within-patient ordering setting 3L + r.
We apologize for the lack of clarity on this setting. It differs from 3L by the ordering loss, by batching (same-patient landmarks share a batch) and by FiLM age encoding. The submitted appendix incorrectly listed a different checkpoint criterion, but this is not the case. In any case, we cannot attribute its accuracy gains to the ordering loss alone, and we say so in both the discussion (Section 5) and Appendix H.
W2 / Q2. One fixed random landmark per patient, and the latest-landmark evaluation.
Thank you for suggesting this control, which we found very informative. We trained 1Lrand for GRU, Transformer and MOTOR-Probe on all eight tasks (Table 9, Appendix E.3). The landmark is drawn once from each patient’s 3L set and held fixed, so 1Lrand and 3L share the same landmark distribution. GRU-ODE runs are slower and will be added to the final version.
Placement (1Lrand − 1L) improves IBS in 23/24 configurations but C-index in only 15/24. Volume (3L − 1Lrand) further improves C-index in 23/24 and IBS in 22/24. Much of the IBS gain comes from placement (e.g., GRU on MIMIC-IV), while the C-index gain comes mainly from supervising several landmarks. Both where and how often a trajectory is supervised therefore matter, and this result was added to Section 4.1.
Regarding the latest-landmark pool, we now also report all models under the latest-landmark evaluation (Table 7), which fits the training distribution of 1L. 3L still improves C-index in 36/48 and IBS in 40/48 configurations. Further patient-level bootstrap intervals (Table 17) confirm the IBS gain on this pool (17/24 configurations) but show that the C-index gain is mostly within sampling uncertainty. We have added this comparison to Section 4.1.