- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper addresses an important practical problem: event-prediction evaluations can yield artificially inflated performance by exploiting temporal or unit-specific information rather than genuine predictive signals. To detect such shortcuts, the authors propose two diagnostic tests conducted before training event-prediction models: a temporal test that determines which of two windows occurs later, and an identity test that assesses whether two windows matched on time. They evaluate these tests across six public domains: mortgages, ICU records, hard drives, wind turbines, student clickstreams, and vessels. A particularly interesting finding is that carry-forward imputation can exacerbate performance inflation.
Critical strengths
- The problem is practical: event prediction can be 'reward hacked' by recognizing a window or unit, instead of learning useful forecasts
- The findings are useful: a carry-forward imputation makes ICU patients easier to re-identify, therefore inflating performance by recognizing patients
Critical weaknesses
- This work has limited methodology novelty: both tests are built on established classification-based tests and verification tests. Moreover, the empirical contribution needs stronger justification, what evaluation decision should practitioners make given this test results?
- This works provides weak connection to prospective event prediction. Section 8 provides an informative bridge experiment. But it does not address whether the authors' method identify misleading performance in realistic forecasting tasks.
Questions for authors
- Given the problem you brought, can you provide solutions to mitigate the effect, i.e as a healthcare practitioner, how can I avoid this inflation? what is benefit of establishing leakage controls.
- The bridge experiment does not fully prove that unit recognition is the only cause for performance. Does this gap persist with other scenarios, such as fixed test windows and the same training-row counts?
- The drive experiment compares devices of the same model. But the ICU experiment did not. How much identity exposure remains when both windows come from the same site?
2 comments
Thank you for the detailed response and answering my question.
But I still have some reservations about the novelty of the overall contribution. I remain uncertain about how reliably the audits would flag misleading performance across other forecasting settings.
Response to Gkix: controlled forecasting, hospital matching, and mitigation
Q1. Mitigation. For new-patient deployment, evaluate and select preprocessing on held-out patients, including within cross-validation. For known-patient forecasting, use only training labels resolved before prediction. Audit each preprocessing variant to identify comparisons worth investigating, and include time and measurement-count baselines on the same cohort. Carry-forward retains a 0.046 advantage over raw in the held-out-patient comparison in Table 5; random patient-hour overlap makes that advantage appear 0.163. That is the benefit of the controls: the held-out comparison measures the gain on unseen patients.
Q2. Fixed test windows and equal training counts. Every controlled overlap comparison uses identical test rows and matched per-class training counts. The carry-forward-minus-raw difference in overlap gaps is +0.103 [+0.091, +0.114] in sepsis case-control classification, +0.117 [+0.102, +0.131] in sepsis forecasting, and +0.083 [+0.068, +0.099] on PhysioNet 2012. A separate size control changes AUC by at most 0.0022. The manuscript reports the controlled design.
Forecasting. Section 5 uses pre-onset features to predict onset within 12 hours. The complete-follow-up and within-patient chronological comparisons are in Section 5.2 and Table 6. The latter finds no positive mean overlap gap for any fill; carry-forward gives −0.007 [−0.014, +0.001]. These are protocol comparisons, not a decomposition attributing the entire gap to unit recognition. The random-split exaggeration, the chronological control and the hour-of-stay baseline each repeat within each hospital system analyzed alone (Appendix H, Table 13).
Q3. Within one hospital. Hospital matching leaves carry-forward identity AUC at 0.989, versus 0.993 with unmatched hospitals. The corresponding values are 0.971 versus 0.978 for model fill, 0.920 versus 0.946 for six-hour carry-forward, and 0.879 versus 0.916 for raw values. Substantial discrimination remains within hospital.
Novelty and the rewritten presentation are addressed in the general response; the revised PDF replaces metaphorical terms with the classification tasks they denote.