acceptodds
Withdrawn from ICLR 2027

Measuring the Temporal and Identity Readability of Operational Logs Across Six Domains

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Retrospective event-prediction evaluations can reward recognition of temporal position or unit identity, making apparent predictive performance difficult to interpret. We introduce a reusable audit that measures these signals in operational logs before an event-prediction model is evaluated. Using windows from units without recorded adverse events, the audit asks two questions: which of two windows from the same unit came later, and whether two windows matched on time and offset came from the same unit. These tasks measure temporal readability and identity exposure, respectively; under their stated exchangeability nulls, expected AUC is exactly 0.5. We release both audits as one harness, driven by a small adapter and a declaration file, and apply it across six public domains. Removing amortization clocks reduces temporal AUC in mortgage records from 0.834 to 0.549, while identity AUC stays at 0.994. In ICU records, measurement counts alone reveal temporal position (AUC 0.634) and distinguish same-patient from different-patient pairs (AUC 0.879). A controlled case-control experiment connects identity exposure to evaluation inflation: carry-forward imputation increases the AUC gap between overlapping-unit and unit-disjoint splits by 0.118. This experiment measures susceptibility to unit overlap rather than prospective prediction skill. Together, the audits characterize temporal and identity signals that can confound evaluation and motivate controls appropriate to the intended prediction task.

Primary area:
datasets and benchmarks
Keywords:
evaluation, benchmarks, negative controls, operational logs, identity exposure, temporal readability, time series

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GkixOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses an important practical problem: event-prediction evaluations can yield artificially inflated performance by exploiting temporal or unit-specific information rather than genuine predictive signals. To detect such shortcuts, the authors propose two diagnostic tests conducted before training event-prediction models: a temporal test that determines which of two windows occurs later, and an identity test that assesses whether two windows matched on time. They evaluate these tests across six public domains: mortgages, ICU records, hard drives, wind turbines, student clickstreams, and vessels. A particularly interesting finding is that carry-forward imputation can exacerbate performance inflation.

Critical strengths
  1. The problem is practical: event prediction can be 'reward hacked' by recognizing a window or unit, instead of learning useful forecasts
  2. The findings are useful: a carry-forward imputation makes ICU patients easier to re-identify, therefore inflating performance by recognizing patients
Critical weaknesses
  1. This work has limited methodology novelty: both tests are built on established classification-based tests and verification tests. Moreover, the empirical contribution needs stronger justification, what evaluation decision should practitioners make given this test results?
  2. This works provides weak connection to prospective event prediction. Section 8 provides an informative bridge experiment. But it does not address whether the authors' method identify misleading performance in realistic forecasting tasks.
Questions for authors
  1. Given the problem you brought, can you provide solutions to mitigate the effect, i.e as a healthcare practitioner, how can I avoid this inflation? what is benefit of establishing leakage controls.
  2. The bridge experiment does not fully prove that unit recognition is the only cause for performance. Does this gap persist with other scenarios, such as fixed test windows and the same training-row counts?
  3. The drive experiment compares devices of the same model. But the ICU experiment did not. How much identity exposure remains when both windows come from the same site?
2 comments
AuthorsOpenReview ↗

Response to Gkix: controlled forecasting, hospital matching, and mitigation

Q1. Mitigation. For new-patient deployment, evaluate and select preprocessing on held-out patients, including within cross-validation. For known-patient forecasting, use only training labels resolved before prediction. Audit each preprocessing variant to identify comparisons worth investigating, and include time and measurement-count baselines on the same cohort. Carry-forward retains a 0.046 advantage over raw in the held-out-patient comparison in Table 5; random patient-hour overlap makes that advantage appear 0.163. That is the benefit of the controls: the held-out comparison measures the gain on unseen patients.

Q2. Fixed test windows and equal training counts. Every controlled overlap comparison uses identical test rows and matched per-class training counts. The carry-forward-minus-raw difference in overlap gaps is +0.103 [+0.091, +0.114] in sepsis case-control classification, +0.117 [+0.102, +0.131] in sepsis forecasting, and +0.083 [+0.068, +0.099] on PhysioNet 2012. A separate size control changes AUC by at most 0.0022. The manuscript reports the controlled design.

Forecasting. Section 5 uses pre-onset features to predict onset within 12 hours. The complete-follow-up and within-patient chronological comparisons are in Section 5.2 and Table 6. The latter finds no positive mean overlap gap for any fill; carry-forward gives −0.007 [−0.014, +0.001]. These are protocol comparisons, not a decomposition attributing the entire gap to unit recognition. The random-split exaggeration, the chronological control and the hour-of-stay baseline each repeat within each hospital system analyzed alone (Appendix H, Table 13).

Q3. Within one hospital. Hospital matching leaves carry-forward identity AUC at 0.989, versus 0.993 with unmatched hospitals. The corresponding values are 0.971 versus 0.978 for model fill, 0.920 versus 0.946 for six-hour carry-forward, and 0.879 versus 0.916 for raw values. Substantial discrimination remains within hospital.

Novelty and the rewritten presentation are addressed in the general response; the revised PDF replaces metaphorical terms with the classification tasks they denote.

Thank you for the detailed response and answering my question.

But I still have some reservations about the novelty of the overall contribution. I remain uncertain about how reliably the audits would flag misleading performance across other forecasting settings.

Reviewer_ooLhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

In this paper authors inverstigate properties of datasets as distinguishible in time and indetifyability. They develop test their audit software on 6 real datasets to detect those properties. A model that can be applied accross the datasets is a contribution. Authors also find that filling missing values improves identyfyablity in one of the datasets.

Critical strengths

Multiple datasets, multiple domains. Authors peformed imputation experiment which is nice

The developed software is reusable and potentially useful for other inverstiagtors

Critical weaknesses

It is unclear to me how this can be used in practice. Altough the imputation experiment is nice to show, it only measures one way of data imputation and therefore is has limited generalizability.

The novelty is limited. There are other methods out there that do simialr tests and it is unclear to me how this approach push the filed forward. This comments is also related to my comment 1. Since I do not cklearly see practical applicaitons it is difficuly for me to even image an experiment where it can be shown that this method is better than others.

Questions for authors

It would be great if authors provide some clues how this can be used. If I have my own dataset and run heir software and get 0.6 and 0.7. What does this mean to me. What is the practival use?

Typically in most real datasets it will be rather easy to detemine if they come from different times as typically conditions of data acqusition change over time and that can be detected. So if I detect something like that what usuful informaiton I gain? It would strengthen the paper to demonstrate how the audit distinguishes expected, harmless variation from information that produces misleading evaluation results, and what useful decisions its scores support beyond established evaluation practices.

1 comment
AuthorsOpenReview ↗

Response to ooLh: practical use and imputation comparisons

Q1. “I get 0.6 and 0.7.” These scores indicate that the fitted probes distinguish temporal order and same-unit pairs above chance. They are not event-prediction scores. Choose the split from deployment regardless of these values, then compare scores across preprocessing and feature ablations to select follow-up checks. Section 6 gives the procedure and a worked sepsis example.

Q2. Harmless versus misleading information. The chronological control supplies a direct comparison: carry-forward identity remains high, but the positive gap seen with random patient-hour splits is absent when training uses only earlier, resolved labels. Clickstreams provide another contrast: identity AUC is 0.660, with an overlap gap of +0.002 [−0.003, +0.006]. The audit locates information; the downstream comparison establishes its consequence. For timing specifically, the temporal audit shows that ICU measurement counts encode time in stay. On the complete-follow-up sepsis cohort, hour of stay alone outperforms raw values plus counts (0.728 versus 0.681). This motivates a time-only baseline to establish whether clinical features improve prediction beyond temporal position.

Imputation breadth. We compare five variants on sepsis and four on PhysioNet 2012. Raw and mean fill have similar identity scores and overlap gaps, while longer carry-forward raises both in the unit-overlap comparisons. On PhysioNet 2012, the carry-forward increase over raw is +0.083 [+0.068, +0.099] on the overlap gap.

Novelty and the rewritten presentation are addressed in the general response; the revised PDF replaces metaphorical terms with the classification tasks they denote.

Reviewer_VnZKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper describes a dataset-side audit with two classifier-based tests on windows from units without adverse events. The temporal test asks which of two windows came later, the identity test asks whether two windows matched on time and offset come from the same unit. Under exchangeability nulls the expected AUC is exactly 0.5 (Prop. 1). The audit is done on six public domains (mortgages, ICU sepsis, Backblaze drives, wind SCADA, OULAD, AIS vessels), though three of them are near the 0.5 number (wind, vessels, student clicks).

Critical strengths
  • temporal and identity leakage in event-prediction evaluation is a real problem
  • statistics are careful: forced-null proposition, unit-disjoint splits, a dyadic cluster bootstrap and the clock-removal tier ladder. Limitations are stated well
  • some individual findings are practically interesting, e.g. ICU measurement counts reveal temporal position (AUC 0.634) and identity (0.879)
Critical weaknesses
  • unclear motivation - why is an additional readability measure necessary, and what does it tell a practiioner?
  • unclear contribution and limited insight, many values and curves, no generalization and guidance
  • presentation is often metaphorical, not precise ("AI style"). A direct explanation of problem, method, and some specific takeaways would be very helpful
  • limited methodological novelty (test are classifier-based two-sample tests (Lopez-Paz & Oquab 2017)
Questions for authors
  • What decision should a practitioner take based on the audit that they would not already take by using unit-disjoint, time-respecting splits?
  • For one or more domains, pleas show a realistic forecasting protocol in which the audit values predict the gap between leaky and leak-free evaluation?
  • Which findings generalise beyond the six datasets, and what should the reader remember from the paper?
1 comment
AuthorsOpenReview ↗

Response to VnZK: practical decisions, forecasting, and generalization

Q1. What decision changes beyond careful splits? The audit does not replace split selection. It helps choose what to check:

Preprocessing choice: a shared-patient comparison can mislead preprocessing selection. In sepsis forecasting, unlimited carry-forward leads model fill by 0.079 AUC with shared patients; with patients held out the two are level (0.006), and model fill leads six-hour carry-forward by 0.028 (Section 5.1, Table 5). The paired difference in overlap gaps (model fill minus six-hour carry-forward) is −0.035 [−0.048, −0.022]. On mortality, carry-forward's advantage over raw values is likewise exaggerated: 0.150 with shared patients versus 0.068 with patients held out. The audit identifies comparisons to investigate; evaluation under the deployment split determines the choice.

Baselines: the temporal audit detects timing information in measurement counts. On the complete-follow-up sepsis task, hour of stay alone outperforms raw values plus counts, 0.728 versus 0.681.

Offsets: drive temporal AUC rises from 0.525 at 7–29 days to 0.734 at 90–180 days. Measurements should use offsets relevant to the intended prediction setting.

Q2. Forecasting and the evaluation gap. Section 5 predicts onset within 12 hours. Under random patient-hour overlap, raw, six-hour and unlimited carry-forward have identity AUCs 0.916, 0.946 and 0.993 and overlap gaps +0.031, +0.110 and +0.147. Requiring complete follow-up preserves the carry-forward-minus-raw gap contrast at +0.117 [+0.104, +0.130]. Under the within-patient chronological control, no fill has a positive mean overlap gap (Table 6). Identity AUC flags recognizability; its evaluation consequence depends on the protocol. It does not estimate gap magnitude.

Q3. What generalizes? The preprocessing effect holds on two ICU datasets and outcomes, and the forecasting results hold within each of the two sepsis hospital systems separately (carry-forward minus raw gap +0.138 and +0.140; hour of stay alone above raw values in both). The broader survey shows that timing and identity can occupy different features: removing mortgage clocks reduces timing discrimination while leaving identity high. The imputation comparison establishes an effect within ICU data; the 0.9 division in Table 3 is descriptive, not a universal threshold. Section 6 gives the practical procedure.

Presentation and novelty. We replaced metaphorical terminology with the classification tasks and their measured consequences. The introduction states what is new: not a test, but a measurement of identity and timing signal as dataset properties with an exact chance level (AUC 0.5) under stated nulls, which to our knowledge has not been reported for both together across domains. The appendix retains the statistical controls and concentrates the related-work discussion on close methods.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: practical value and new experiments

Thank you. All three reviews ask what the audit adds beyond careful evaluation splits. The split follows from deployment, regardless of the audit score. The audit identifies features and preprocessing choices whose consequences should be checked under that split.

A shared-patient comparison overstates carry-forward's benefit 3.5 times. Carry-forward raises identity AUC from 0.916 to 0.993. In 12-hour sepsis forecasting, its advantage over raw values is 0.163 AUC when patient hours are randomly divided between training and test, versus 0.046 with patients held out. Test rows and per-class training counts are identical between arms.

The effect replicates. On PhysioNet 2012, with hospital mortality as the outcome, carry-forward increases the overlap gap over raw values by +0.083 [+0.068, +0.099]. This is a second dataset and outcome; it shares a source hospital with the sepsis dataset.

High identity AUC does not imply inflation under every protocol. With complete follow-up and training restricted to earlier patient hours whose labels have resolved, no fill has a positive mean overlap gap. Carry-forward gives −0.007 [−0.014, +0.001]. This control supplies only earlier negative hours, changing both the information and amount of patient overlap.

The clock alone beats the raw clinical values. On the complete-follow-up cohort, hour of stay alone scores 0.728 on held-out patients, versus 0.681 for raw values plus counts: a paired difference of +0.046 [+0.023, +0.070].

The tests are established tools. What is new is the measurement: identity and timing signal measured together as dataset properties, without specifying a downstream prediction target. To our knowledge, no earlier work does this on the same panels across domains. We then test their consequences under controlled evaluation protocols.

The revised PDF adds the second ICU dataset (Section 4), vessels and mean and model fills in the overlap comparison (Section 4), forecasting with chronological and clock-baseline controls (Section 5), hospital-matched analyses (Appendix H), and practical guidance (Section 6). We also rewrote the methods directly, without metaphor.

AuthorsOpenReview ↗

Revision update: Figure 1 and manuscript source added

We uploaded an updated PDF and supplementary zip.

Figure 1 on page 2 illustrates the two pairing audits and the controlled overlap design. It also shows the overlap gaps for all 13 rows of Table 3 beside their identity AUCs.

The supplementary zip now includes the auditable manuscript source. Reported AUCs, gaps and intervals are inserted through macros generated from the included result files. The included audit script checks for hand-typed numerical results and for macros that no longer match the stored results, and the README explains how to run it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.