acceptodds
Under review as a conference paper at ICLR 2027

Sparse Multi-Landmark Supervision for Longitudinal EHR Survival Prediction

Abstract

Electronic health records (EHRs) accumulate irregularly over years, yet survival models that support longitudinal risk prediction are commonly trained at only one prediction time per patient and then queried repeatedly as new clinical observations arrive. We study sparse multi-landmark supervision, which trains from several points along each patient trajectory, as a model- and objective-agnostic strategy to reduce this mismatch. Across eight tasks, two health systems, and six model families, training with three landmarks improves integrated Brier score in all 48 task-model configurations and concordance index in 45 of 48 under multi-landmark evaluation. These gains persist when optimizer updates or input tokens are matched, when recent clinical history is progressively truncated, and across retraining. Multi-landmark supervision also weakens coupling between predicted risk and EHR recording patterns (e.g., code density and redundancy). On EHRSHOT, three-landmark training can further compensate for substantially fewer training patients. Beyond predictive performance, we introduce trajectory-aware evaluation for models used repeatedly over time. Multi-landmark supervision makes successive predictions more temporally consistent, while direct trajectory analysis reveals that consistency and informative risk evolution are distinct properties. A ranking-augmented proof of principle shows that explicitly relating predictions across landmarks can alter trajectory direction. In a nutshell, our results show that better use of existing longitudinal records can substantially improve survival prediction, while motivating models and evaluation that explicitly account for how risk evolves within patients over time.

Primary area:
learning on time series and dynamical systems
Keywords:
electronic health records, survival analysis, landmarking, dynamic prediction, irregular time series, healthcare

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_qL2WOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an add-on training strategy that supplements existing longitudinal EHR survival models without changing their architecture or survival objective. It addresses a mismatch between training at one prediction time per patient and making repeated predictions during follow-up. Instead of supervising only the latest eligible prediction time, called one landmark training (1L), the approach supervises three selected times along each patient’s record, called three landmark training (3L). At each landmark, the model uses only the available history and predicts the remaining time until the event or censoring. Validation covers eight tasks across EHRSHOT and MIMIC-IV, six model families, and five development folds. The plain 3L results improve integrated Brier score in all 48 configurations and concordance in 46 of 48. Additional experiments examine training budgets, cohort size, record truncation, and prediction variability. The paper also evaluates successive predictions within individual patients, showing that better temporal agreement does not necessarily imply more informative risk trajectories.

Critical strengths
  1. The main strength is that the proposed strategy can supplement existing survival models without requiring a new architecture or survival objective. Its benefits are demonstrated across two different EHR datasets and six model families, including models trained from scratch and a probe using frozen pretrained representations. This breadth supports a practical contribution: researchers can improve how they use existing patient histories while retaining their established modeling approach. The plain 3L results in Table 5 provide substantial evidence for this benefit.

  2. The paper examines longitudinal prediction behavior beyond average discrimination and prediction error. It distinguishes agreement between successive predictions from changes in forecast hazards and within-patient risk direction. The finding that improved agreement can coexist with flatter hazards and weak separation between case and control risk slopes is particularly useful. These analyses reveal limitations that standard survival metrics would miss and make the paper informative beyond its central training recommendation.

Critical weaknesses
  1. The primary results do not consistently identify which configuration is being evaluated. The MOTOR-Probe gains in Tab. 1 correspond to the exploratory "+r" configuration in Table 5, rather than plain three landmark training. For example, the AMI concordance gain is reported as 16.46 percentage points, whereas plain three landmark training improves it by 5.80 points. Appendix H states that the augmented configuration also changes batching, age conditioning, and checkpoint selection. Its gains therefore cannot be attributed to additional landmark supervision alone. Figure 12 and Table 13 also report different five-year slope AUCs for apparently identical MOTOR-Probe configurations. For pancreatic cancer, the respective baseline and three landmark values are 0.493/0.542 and 0.432/0.448. These discrepancies affect central results and require reconciliation. The broad benefit in Table 5 remains, but confidence in the reported effect sizes and trajectory conclusions is reduced.

  2. The experiments change both the number and placement of supervised prediction times. The baseline always uses the latest eligible landmark, whereas three landmark training includes earlier portions of the trajectory. The results therefore establish an advantage over latest-only training, but do not show whether several landmarks outperform one representative landmark per patient. A fixed randomly selected landmark would be the most useful missing control. The budget and uncertainty claims also need qualification. Optimizer updates are explicitly matched for MOTOR-Probe, while matching nominal input tokens for other encoders does not consistently match total training computation. The reported configuration counts indicate positive mean differences, rather than independently established improvements. Because the development folds overlap and the test population is fixed, variation across folds does not capture uncertainty from sampling different test patients.

Questions for authors
  1. Which configuration produced the MOTOR-Probe row in Tab. 1 and each primary figure? Please provide the primary comparison using plain three landmark training and reconcile Fig. 12 with Tab. 13, including any differences in models, patients, or metric aggregation.

  2. How does three landmark training compare with one landmark per patient selected uniformly from the ten landmark pool and then held fixed, using the same optimizer budget? This experiment would distinguish the benefit of landmark placement from the benefit of supervising several histories from each patient. Including a representative encoder trained from scratch would also strengthen the budget comparison.

  3. Are landmark pairs from the same patient included in pooled concordance calculations? Please provide confidence intervals for the principal performance differences using patient-level resampling. The unusually high and variable integrated Brier scores in several configurations also need explanations and comparison with a simple survival reference evaluated with the same observations.

  4. What justifies interpreting increasing fixed-horizon risk among eventual cases as evidence of better temporal prediction? Remaining event-free and receiving new information can legitimately lower predicted risk, even for someone who later experiences the event. Similarly, informative new observations can justify disagreement with an earlier prediction transported forward in time. Please clarify the interpretation of these diagnostics and the rationale for encouraging increasing risk through the ranking objective.

3 comments
AuthorsOpenReview ↗

Addressing results discrepancies, weaknesses and questions (part I)

Thank you for your very careful review. You were right about the reporting discrepancies, and we sincerely apologize for them. We are grateful that you caught them, as correcting them made the paper more reliable. They came from tagging errors in our table-building scripts and are detailed under “Correction of reported results” in the general comment. For readability, all updated and additional elements are highlighted in blue in the currently revised manuscript.

W1 / Q1. Which configuration are reported, and former results mismatch.

i) On main Table 1 values.

The MOTOR-Probe row reported 3L + r. It now reports plain 3L (e.g., AMI C-index +5.87 pp). No other row was affected, and all primary figures already use plain 3L. The MOTOR-Probe gain remains, but is reduced compared to MOTOR-Probe + r. The updated overall gains have been corrected in Abstract and Section 4.1 (e.g., "median gains of 3.54 and 3.48 percentage points (pp)" instead of "4.34 and 3.48 pp" previously).

ii) On MOTOR-Probe 3L values.

Tables 2, 6 and 16, and Figure 6 used the update-matched run. They now use standard 3L training, while the update-matched variant appears only in Table 12. The update-matched run was a slightly more conservative version of the probe, but differences with the updated values are small (e.g., AMI C-index 63.30 vs. 63.38). This error did not overturn any claim.

iii) On SlopeAUC mismatch.

The EHRSHOT slopes in the former Table 13 (now Table 16) were computed on a position-stratified test pool instead of the time-stratified pool used elsewhere for this dataset. After recomputation, Table 2, Figure 6 and Table 16 agree (pancreatic cancer: 0.493 / 0.539 / 0.611 for 1L / 3L / 3L + r).

iv) Regarding the within-patient ordering setting 3L + r.

We apologize for the lack of clarity on this setting. It differs from 3L by the ordering loss, by batching (same-patient landmarks share a batch) and by FiLM age encoding. The submitted appendix incorrectly listed a different checkpoint criterion, but this is not the case. In any case, we cannot attribute its accuracy gains to the ordering loss alone, and we say so in both the discussion (Section 5) and Appendix H.

W2 / Q2. One fixed random landmark per patient, and the latest-landmark evaluation.

Thank you for suggesting this control, which we found very informative. We trained 1Lrand1L_{rand} for GRU, Transformer and MOTOR-Probe on all eight tasks (Table 9, Appendix E.3). The landmark is drawn once from each patient’s 3L set and held fixed, so 1Lrand1L_{rand} and 3L share the same landmark distribution. GRU-ODE runs are slower and will be added to the final version.

Placement (1Lrand1L_{rand} − 1L) improves IBS in 23/24 configurations but C-index in only 15/24. Volume (3L − 1Lrand1L_{rand}) further improves C-index in 23/24 and IBS in 22/24. Much of the IBS gain comes from placement (e.g., GRU on MIMIC-IV), while the C-index gain comes mainly from supervising several landmarks. Both where and how often a trajectory is supervised therefore matter, and this result was added to Section 4.1.

Regarding the latest-landmark pool, we now also report all models under the latest-landmark evaluation (Table 7), which fits the training distribution of 1L. 3L still improves C-index in 36/48 and IBS in 40/48 configurations. Further patient-level bootstrap intervals (Table 17) confirm the IBS gain on this pool (17/24 configurations) but show that the C-index gain is mostly within sampling uncertainty. We have added this comparison to Section 4.1.

AuthorsOpenReview ↗

Addressing weaknesses and questions (part II)

W2. Budget and uncertainty claims.

We agree with all three points.

  • (i) Optimizer updates are matched only for MOTOR-Probe, and matching input tokens does not match total training compute (Table 11). Section 4.2 now says so.
  • (ii) Configuration counts reflect the sign of mean differences, not independently established improvements and we clarify this in Section 4.1.
  • (iii) We added 95% patient-level bootstrap intervals for the 3L − 1L difference (Tables 17-18). Patients are resampled 1,000 times with all their landmarks, paired across models and folds, for GRU, Transformer and MOTOR-Probe on all eight tasks. The remaining three model families will be added to the final version.

Q3. Concordance, confidence intervals and high IBS values.

Same-patient pairs.

They are included in our C-index computations, but make up at most 0.12% of comparable pairs. Excluding them changes C-index by at most 0.04 pp and the 3L − 1L difference by at most 0.012 pp, with no configuration changing sign. We clarify this in Appendix C.1.

Confidence intervals.

On the ten-landmark pool, the interval excludes zero in favor of 3L in 21/24 configurations for IBS and 16/24 for C-index, and never in favor of 1L (Tables 17-18). On the latest-landmark pool, the IBS gain remains clear (17/24), but the C-index gain is mostly within sampling uncertainty (8/24 in favor of 3L, 1/24 in favor of 1L). These intervals capture test-patient sampling given the trained models, while the fold standard deviations capture training variability. We report both.

Configurations whose 95% patient-level bootstrap CI of the 3L − 1L difference excludes zero. GRU, Transformer and MOTOR-Probe on eight tasks (24 configurations per pool). Full intervals in Table 17.

MetricTen-landmark poolLatest-landmark pool
C-index16/24 favor 3L · 0 favor 1L8/24 favor 3L · 1 favors 1L*
IBS21/24 favor 3L · 0 favor 1L17/24 favor 3L · 0 favor 1L

* Transformer on AMI (−4.04 pp).

High IBS.

We agree that some IBS values remain unusually high, for instance for all models on HPT and for DeepHit/Dynamic-DeepHit on AMI (EHRSHOT) under the ten-landmark evaluation. A sanity check showed that such configurations do not beat a Kaplan-Meier reference (fitted on 3L training rows) under 1L or 3L, although 3L moves them closer. This points to poorly calibrated absolute risk, most severely on EHRSHOT HPT, which we will investigate further. The 3L − 1L IBS gains there therefore reflect reduced miscalibration, not well-calibrated absolute risk. We state this in Section 4.1. C-index results and the paired 3L − 1L contrast are unaffected.

Q4. Interpreting rising risk and TV.

We agree. Remaining event-free and new information can legitimately lower a case’s predicted risk and can justify disagreement with a transported prediction. We therefore treat neither rising case risk nor low TV as a target. Both are descriptive diagnostics, compared between 1L and 3L on identical patients and pairs. SlopeAUC is more informative because it contrasts cases with controls, who are subject to the same effects (Section 3.3, Appendix C.2). The ordering loss does assume that a case’s short-term risk rises before the event. We now state this assumption explicitly (Appendix H) and present the loss only as a proof of principle that trajectory direction can be shaped. In our experiments, it coincides with better case-control separation (Table 2). The assumption is task-dependent: recurrent or relapsing conditions call for learning risk dynamics directly rather than imposing monotonic increase. This discussion has been added to Section 5.

Dear Reviewer qL2W, we hope you don’t mind a brief follow-up before the discussion period ends. Following your review, we corrected the reporting errors and re-verified all tables, added the random-landmark control you suggested ($1L_{rand}$), and reported patient-level bootstrap intervals. We would value your feedback on whether these changes resolve your concerns, and we would be happy to answer any remaining question while there is still time. If you find our answers satisfactory, we would be grateful if you would consider updating your assessment. Thank you again for the care you put into this review, and for spotting the discrepancies in our reported results.

Reviewer_gRjZOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors motivated the paper to addresses a mismatch in EHR survival modeling where models are typically trained at a single prediction time per patient but deployed repeatedly throughout longitudinal follow-up. To solve this the authors proposed a sparse multi-landmark supervision framework as a model- and objective-agnostic strategy to align training with deployment. They evaluated the framework across multiple tasks, across health systems and model families and the experimental results indicate that the approach improves predictive accuracy, robustness to record perturbations, and temporal consistency. Further, the authors added a trajectory-aware evaluation framework and to demonstrate the alignment gap between consistency and real-world use cases. A proof-of-concept shows that explicit cross-landmark objectives can alter trajectory direction.

Critical strengths
  • The train-deploy mismatch in longitudinal EHR modeling is often known but nevertheless not considered for practical purposes. Framing the solution around supervision strategy rather than architecture or representation is a smart, practical contribution that aligns well to further improvements and centralization of the key claim

  • The authors presented experimetns with a detailed control design. Of special note the efforts to martch optimizer update and input-token budges and multi-landmark test pool for evaluation. Continuing on this trend, they presented a thoughtful analysis of the trajector aware evalaution and suplanted these with a clear conceptual distinction to distinguish repeated supervision with relational supervision

  • The paper had enough details spanning data generation, training mechanisms, and metric definisionts - with the authors committment to release the code publicly, the paper should act as a springboard for further research

Critical weaknesses

However, ther are a few aspects that might warrant further discussion

  • The eligibility criteria seems to be biased towards chroninc cases and excludes patients with acute trajectories such as those during ICU/emergency care. This might limit the scope of the analysis and if not applicable should be clarified
  • The Results performance may indicate that model's capacity to learn monotonic risk progression is limited. IT would be interesting to analyze where this loss is coming from
  • The trajectory dynamics seems to be a rather important contribution and this objective should be evaluated alongside the main results

Some other comments

  1. Add a brief ablation on L=2, 4, 5 to justify the choice of L=3.
  2. Discuss how eligibility criteria (≥3 codes, ≥3 months gap) affect applicability to acute/short-stay populations.
  3. Clean mathematical notation for publication readiness
Questions for authors

See above

2 comments
AuthorsOpenReview ↗

Addressing main weaknesses and comments

Thank you for your positive review and helpful suggestions. For readability, all updated and additional elements are highlighted in blue in the currently revised manuscript.

Eligibility and acute or short-stay populations.

Our setting is risk prediction months to years ahead, hence the three-month gap and the minimum of three codes. Short records are not excluded as such: a patient is included as soon as one landmark meets both criteria, and MIMIC-IV records span a median of only 0.1–2 years (Table 3). Acute prediction over hours or days is indeed outside our scope as we focus on mid- to long-term tasks. It would require shorter gaps, finer time encoding and different hazard intervals. We also expect landmark selection to matter more when histories and residual times are both short. We now state this as a limitation (Section 5, “Limitations”).

## Limited monotonic risk progression. We extended the trajectory analysis to all six model families (Table 16). Under 1L, the median case slope is negative in 41/48 configurations. Under 3L it is positive in 22/48, mostly for DeepHit and Dynamic-DeepHit (15/22). Our reading is that the limitation comes from the objective: each landmark is supervised independently, so nothing relates successive predictions for the same patient. Adding a within-patient ordering loss (3L + r) yields positive case slopes in 25/32 configurations (the ordering loss is not applied to DeepHit and Dynamic-DeepHit), which is consistent with this reading. Future work could learn risk dynamics directly, for example with neural-ODE or flow-matching approaches [1]. We clarified this in Section 5 and Appendix H.

[1] Estévez, et al., “Learning Continuous Patient Trajectories from Electronic Health Records.“ arXiv preprint arXiv:2609.36144 (2026).

Trajectory objective alongside the main results.

Table 2 (main text) now reports slopeAUC under 1L, 3L and 3L + r on all eight tasks: 3L improves on 1L in 6/8 tasks, and 3L + r is best in 7/8. Tables 6–7 report the predictive performance of 3L + r for the from-scratch encoders and MOTOR-Probe. Since 3L + r also changes batching and age encoding, we present it as a proof of principle and suggest further work on learning within-patient dynamics (Appendix H, and updated Section 5).

Ablation on the number of landmarks L.

Table 8 now covers L∈[1,2,3,4,5,10]L \in [1, 2, 3, 4, 5, 10 ]. Most of the IBS gain is reached by L = 2-3, while C-index keeps improving up to L = 5-10. L = 3 is a conservative choice on this plateau. This is clarified in Appendix E.2.

Notation.

We unified the notation (landmark set, a single survival notation, consistent loss names) and added explicit definitions of the piecewise-exponential head, total-variation distance, hazard dispersion and the ordering loss (Section 3.1; Appendices A.3, C, H). We believe this has improved the paper for better readability, and hope you will agree.

Dear Reviewer gRjZ, we hope you don’t mind a brief follow-up before the discussion period ends. Following your suggestions, we clarified the scope regarding acute settings in the new Limitations section, extended the trajectory analysis to all six model families with slopeAUC in the main text, added the ablation on the number of landmarks, and unified the notation. We would be glad to know whether these address your comments, and to answer any remaining question while there is still time. We greatly thank you for your time and supportive review.

Reviewer_nsQ6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates sparse multi-landmark supervision in EHR survival prediction. This paper shows that multi-landmark supervision improves both discrimination and calibration, and it designs a few new metrics (temporal consistency, risk dynamics) and shows that multi-landmark supervision is more consistent and informative.

Critical strengths
  1. The improvement from multi-landmark supervision holds across six model families and eight tasks from two health systems. The method only changes how training examples are built, so it is simple to adopt and a useful basis for more efficient longitudinal training

  2. Beyond standard metrics, the paper evaluates robustness to removed history, coupling to recording patterns, retraining reproducibility and trajectory behavior. It shows that multi-landmark training outperforms single-landmark training in almost all metrics.

Critical weaknesses

W1: The formulation in 3.1 is very similar to Motor [1] and ORA [2], which both use multi-landmark in pretraining. ORA’s ablation studies show that the multi-landmark temporal point process loss outperforms next-token prediction across different models and diverse tasks. This paper seems to apply the same multi-landmark supervision to the downstream stage. It would be better to contextualize this work and discuss its contributions relative to previous papers.

W2: The performance gap between 3L and 1L is not clearly explained. There could be multiple reasons why 3L is better than 1L. (a) The primary evaluation uses the ten-landmark test pool. Its landmarks are much closer in distribution to 3L’s training landmarks than to the single latest landmark 1L trains on. (b) The matched-optimizer control (Appendix B.2) equalizes the number of updates, but 1L then revisits each example about three times as often as 3L does, which could easily lead to overfitting.

W3: It is unclear whether the multi-landmark benefit is scalable to large cohorts and whether it works in low-prevalence tasks (<3% prevalence score). Table 6 shows that 3L training has minimal gains over the 1L on almost all tasks in MIMIC, which has much larger cohorts than EHRSHOT. Also, 3L has the least gain over 1L on low-prevalence tasks in EHRSHOT (PC).

[1] Steinberg, Ethan, et al. "MOTOR: A time-to-event foundation model for structured medical records." International Conference on Learning Representations. Vol. 2024. 2024.

[2] Jing, Zilin, et al. "One loss to rule them all: Marked time-to-event for structured ehr foundation models." arXiv preprint arXiv:2602.00541 (2026).

Questions for authors

Q1: Figure 9 suggests the gain changes little between 50% and 100% of training patients. Is the linear probe already saturated? Could you add 10% and 25%, and also train 1L at the same fractions?

Q2: As line 193 points out, L = 3 captures most of the calibration gain while additional landmarks could further improve distribution search. Could you analyze why calibration saturates earlier than discrimination? Could you discuss the trade-off between the number of landmarks, compute and performance?

Q3: Why should a lower TV distance indicate better temporal consistency? The later prediction conditions on new records, including the landmark event itself. The TV variance is confounded by the different numbers of input events and by how each model responds to different events. Could you justify the metric?

4 comments
AuthorsOpenReview ↗

Addressing main weaknesses

Thank you for your thoughtful review. Your questions helped us clarify what drives the gain. For readability, all updated and additional elements are highlighted in blue in the currently revised manuscript.

## W1. Relation to MOTOR and ORA. Thank you for pointing us to the ORA paper. MOTOR and ORA both use many prediction times per patient in pretraining. We study the downstream stage: how many times each patient is supervised for a given task, whatever the encoder. The two are complementary: even on frozen MOTOR representations, three-landmark training of the probe improves IBS on 8/8 tasks and C-index on 7/8 (Table 1). We added this positioning to Section 2.

W2. What explains the 3L − 1L gap?

(a) Test distribution.

We agree the ten-landmark pool is closer to 3L training. We now also report latest-landmark evaluation, where 1L is tested on its own training distribution (Table 7): 3L still improves C-index in 36/48 and IBS in 40/48 configurations (median +1.72 and +1.31 pp). During training, validation also uses the latest landmark in both settings, which favors 1L. A new control experiment with one random landmark per patient (1Lrand1L_{rand}, Table 9 and reply to Reviewer qL2W) further separates placement from volume: placement alone already helps, and three landmarks help further.

(b) Revisiting examples.

This is a fair point. Under matched updates, 1L revisits each example three times as often, so part of the 3L benefit may act as regularization. We see this as part of the mechanism: 3L exposes more distinct patient states per update. In the primary from-scratch comparisons, both settings use early stopping, so 1L is not forced to over-train. We clarified this in Section 4.2 and Appendix B.2.

W3. Larger cohorts and low prevalence.

Cohort size.

The gain shrinks as the cohort grows but stays positive at full size on most tasks (new Figure 10). On MIMIC-IV, we believe trajectory length matters more than cohort size. Records span a median of 0.1–2 years and 139–296 codes, against 6–8 years and more than 3,000 codes on EHRSHOT (Table 3), so a single 512-code window already covers the full record (Table 4). Consistently, the MOTOR-Probe gain on MIMIC-IV stays small even with 10% of patients (Figure 10). The small MIMIC-IV gains mainly concern MOTOR-Probe as from-scratch encoders gain 0.6–13.4 pp in C-index (Table 1). A likely reason is that MOTOR embeddings already summarize the full history in the context window, so on short records the embeddings at different landmarks are similar.

Prevalence.

Pancreatic cancer (5.2%) is indeed our lowest-prevalence task and shows some of the largest gains: +4.3 to +10.8 pp C-index for the five from-scratch models. Only MOTOR-Probe gains little in C-index (+0.66 pp), but it gains 5.91 pp in IBS. We did not test prevalence below 3%. Combined with short trajectories, such a scenario could make extra landmarks redundant, which we now state as a limitation (Section 5, “Limitations”).

AuthorsOpenReview ↗

Addressing main questions

Q1. Cohort fractions.

We added 10% and 25% as requested, and trained 1L at the same fractions (Figures 9-10). The probe does saturate: gains grow up to 50–65% of patients and then flatten. Mean gain of 3L (pp) per dataset is summarized below:

Average gain when 3L is trained with increasing cohort size (summary of Figures 9 and 10).

DatasetComparisonMetric10%25%50%65%80%100%
EHRSHOTMean gain vs 1L full (100%)C-index−4.55+4.26+3.94+4.00+3.41+5.26
IBS−1.44+0.07+1.00+1.19+1.37+1.42
Mean gain at equal fractionC-index+10.66+9.68+6.36+3.99+5.09+5.26
IBS+12.94+1.66+1.80+1.75+1.58+1.42
MIMIC-IVMean gain vs 1L full (100%)C-index−3.02−1.13−0.23+0.15+0.29+0.60
IBS−5.25−0.28−0.02+0.06+0.07+0.04
Mean gain at equal fractionC-index+0.92+0.61+0.87+0.67+0.54+0.60
IBS−0.10+4.22+0.18+0.11+0.10+0.04

On EHRSHOT, 3L with 25% of patients already matches or exceeds full-cohort 1L. At equal cohort size, the C-index gain doubles as the cohort shrinks from 100% to 10%. We added this to Section 4.2, strengthening our finding that multi-landmarking can be even more beneficial in small-cohort scenarios.

Q2. Number of landmarks.

Table 8 now covers L∈[1,2,3,4,5,10]L \in [ 1, 2, 3, 4, 5, 10 ] with the same number of optimizer updates, so compute is identical across L. IBS saturates at L = 2-3, while C-index keeps improving up to L = 5-10 on several tasks. Our interpretation is that IBS also depends on the overall level of predicted risk. Latest-landmark training sees only a narrow range of short residual times, and a few earlier landmarks are enough to correct this. C-index depends only on ranking, which keeps benefiting from more diverse patient states. The new $1L_{rand} control supports this (Table 9): a single random landmark recovers most of the IBS gain but not the C-index gain. Without an update constraint, cost grows linearly with L, so we chose L = 3 as a conservative trade-off. Our main results likely understate the achievable gain on some tasks. The matched-token control (Table 11) shows that landmarks can also be added at a fixed input budget.

Q3. Why lower TV indicates better consistency.

We agree that new records can legitimately change a prediction, so TV is not expected to reach zero and is not a target in itself. The rationale is that a coherent model’s earlier prediction (conditioned on survival to the next landmark) should equal the average of the later predictions it could make. This idea also underlies temporal-difference survival objectives [1]. Individual pairs will still differ, because new records are informative. TV therefore mixes two things: legitimate updating and incoherence between successive predictions.

We compare 1L and 3L only on identical landmark pairs with identical inputs, so the new information is the same for both models. A lower TV under 3L then means one of two things: 3L contradicts itself less, or it reacts less to new records. TV alone cannot tell them apart, and flat predictions lower it trivially. This is why we report it with hazard dispersion (Figure 5) and trajectory direction (Table 16), and treat all three as diagnostics. We clarified this in Section 3.3 and Appendix C.2.

[1] Maystre & Russo. “Temporally-Consistent Survival Analysis.” NeurIPS 2022

Dear Reviewer nsQ6, we hope you don’t mind a brief follow-up before the discussion period ends. In response to your review, we clarified the position of our work relative to MOTOR and ORA, added latest-landmark evaluation and a random-landmark control $1L_{rand}$ to isolate the 3L − 1L gap, extended the train cohort-size and number-of-landmarks experiments, and clarified the interpretation of TV. We would be glad to know whether these address your concerns, and to answer any remaining question while there is still time. If they do, we would be grateful if you would consider updating your assessment. We sincerely thank you for your time and thoughtful review.

Reviewer_nsQ6OpenReview ↗

Official Comment by reviewer nsQ6

Thanks for the detailed response and most of my concerns are solved. I will keep my current score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment (AC and all reviewers)

We thank the reviewers for their careful and constructive feedback. All three highlight the breadth of the evaluation (eight tasks, two health systems, six model families) and the trajectory-aware analysis. Changes are highlighted in blue in the revised PDF.

1. Correction of reported results (Reviewer qL2W).

Tagging errors in our table-building scripts caused three reporting mistakes for which we sincerely apologize. We re-verified every table and figure, and the headline conclusions are unchanged.

  • Table 1, MOTOR-Probe row reported 3L + r instead of plain 3L. It is corrected, and all other rows are unchanged. The headline result holds: 3L improves IBS in 48/48 configurations and C-index in 46/48 (median gain is now +3.54 and +3.48 pp). These results have been updated in Abstract and results section (Section 4.1).
  • MOTOR-Probe “3L” values in Tables 2 and 6, and Figure 6 came from the update-matched run instead of the full 3L training. They are corrected, and the differences are small (e.g., AMI C-index 63.30 vs. 63.38). The update-matched variant now appears only in Table 12.
  • Former Table 13 (now Table 16): EHRSHOT slopes were computed on a position-stratified test pool instead of the time-stratified pool used everywhere else for this dataset. They are recomputed, and Table 2, Figure 6 and Table 16 now agree.

2. New analyses.

  • Latest-landmark evaluation, where 1L is tested on its own training distribution (Table 7): 3L still improves C-index in 36/48 and IBS in 40/48 configurations.
  • A one-random-landmark control, 1Lrand1L_{rand} , separating landmark placement from landmark volume (Table 9).
  • Additional number of landmarks L∈[1,2,3,4,5,10]L \in [ 1, 2, 3, 4, 5, 10 ] (Table 8).
  • Cohort-size experiments with 10% and 25% fractions, and 1L trained at equal fractions (Figures 9-10). 3L + r reported for all applicable models (Tables 6, 7, 16), with slopeAUC now in the main text (Table 2).
  • Patient-level bootstrap confidence intervals for GRU, Transformer and MOTOR-Probe (Table 17): on the primary evaluation, intervals exclude zero in favor of 3L in 21/24 configurations for IBS and 16/24 for C-index, and never in favor of 1L.

3. Clarifications.

We added a Limitations paragraph (covering acute settings, short trajectories, low prevalence), qualified the budget claims (Section 4.2), clarified the interpretation of TV and risk slopes as diagnostics (Section 3.3, Appendix C), and positioned the work relative to MOTOR and ORA (Section 2). We have also improved and standardized notations.

Renumbering.

Former Tables 5–13 are now Tables 6, 8, 10, 11, 12, 13, 14, 15 and 16, respectively. The former “Figure 12” slopeAUC table is now Table 2, while former Figures 10-11 are now Figures 11–12. Table 7, Table 9 and Figure 10 are new.

Reproducibility.

As stated in the Reproducibility Statement, we will release the full code upon acceptance, including the new experiments and the scripts that generate every table and figure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.