acceptodds
Under review as a conference paper at ICLR 2027

Retrospective Supervision Reshapes EHR Patient States Beyond Future Prediction

Abstract

Electronic health record (EHR) foundation models compress longitudinal records into patient states that are reused across downstream tasks, yet these representations are evaluated mainly by how well they predict future outcomes. This leaves unmeasured how much of the patient's history remains accessible and which patients the representation places near one another. We introduce Chronos, a patient-state model trained with prospective and retrospective objectives: predicting future clinical events and reconstructing a structured fingerprint of prior history. On MIMIC-IV, frozen Chronos states achieve 0.871 mean AUROC across 16 clinical prediction tasks, compared with 0.862 for the strongest existing model, and 0.701 mean AUROC across eight target-masked phenotype-retrieval cohorts, outperforming the best learned baseline by 0.06. In a matched ablation, retrospective supervision leaves future-task performance essentially unchanged while improving retrieval from 0.651 to 0.697 and making both observed and omitted history more recoverable. These effects replicate across independently trained models and persist under projection and record-footprint controls. Our results show that prospective predictiveness, retrospective accessibility, and patient-similarity geometry capture distinct properties of reusable EHR representations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Electronic health records, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jy32OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces CHRONOS, an EHR foundation model pretraining strategy that additionally pretrains by learning to predict prior state. The new pretraining objective does not improve risk prediction but does improve the ability to predict prior state.

Critical strengths

The primary evaluation metric for prospective risk, AUROC on a variety of tasks (as shown in Table 3), is correct, and the overall experimental setup seems roughly correct (although I cannot audit the code, so I can't speak with much confidence here).

Critical weaknesses
  1. This is a very poorly written paper that is difficult to understand. It almost feels entirely AI written. For example, most of the subsections in results do not make any sense or appear to contribute to the claims at all. I don't understand what table 1 or table 2 have to do with the central claims of the paper. This paper needs to be rewritten from scratch by a human to improve clarity.

  2. The baselines don't really make much sense. At minimum, both the prognostic and retrieval experiments should use gradient-boosted tree models. It looks like the retrieval baselines don't include any trained models? Not even a trained logistic regression model?

Questions for authors

Can you release your code so that it can be peer reviewed? This is a paper that introduces a novel algorithm, and then tests it on a new benchmark, comparing to baselines that are also all written by the authors. And the dataset involved here, MIMIC-IV, is very complex. There are so many opportunities for mistakes here, from top to bottom. Due diligence means a code review would be very useful here.

2 comments

Thank you for the comments. We agree that the submission was hard to follow, and we have revised it substantially. We also agree with your point about baselines and have added gradient-boosted trees.

Clarity, and what Tables 1 and 2 have to do with the claims. The central claim is narrow: a patient state's prospective performance does not determine how much of its past history is accessible or which patients are its neighbors. Each table now has a stated role (§5, new overview paragraph):

  • Table 1 is the core experiment. Two CHRONOS models are identical except for the retrospective (fingerprint) loss. History recovery (+0.012 AUROC) and patient retrieval (+0.053 AUROC) improve, at no cost to future-task AUROC (0.869 vs 0.871). That contrast is the paper's claim, and Table 1 shows it on INSPECT as well.
  • Table 2 asks whether the retrieval gain is an artifact. The gain could be a cosine-geometry effect or a record-size effect. It survives train-fitted PCA, and it survives matching patients on record size and on the footprint left by masking.
  • Tables 3 and 4 place the final model among existing models; the paper treats them as context, not as the main evidence.

Concrete changes:

  • §3. Rewritten with a one-line Eq. 1 and plain-language definitions of the two clocks and of the training window. An unused risk definition moved to the appendix.
  • Figures. The results figure moved out of the introduction, and Figure 1 is now the model overview.
  • §5. A short overview paragraph maps each claim to its evidence, and the two slot subsections are merged.
  • Over-qualified prose is cut throughout.

Baselines: gradient-boosted trees for prediction. We added LightGBM on raw code counts from the same input window and rows. It is tuned on validation (learning rate × leaves grid, early stopping) and scored once on test.

  • GBT reaches 0.832 mean AUROC. TF-IDF + logistic regression reaches 0.816, ORA 0.862 and CHRONOS 0.871.
  • In paired, Benjamini–Hochberg-corrected tests, CHRONOS is significantly better than GBT on 14 of 16 tasks. GBT is significantly better on one task (ICU length of stay > 7 days), and it is bold there in Table 3.
  • On INSPECT, an external dataset, CHRONOS also beats GBT on identical rows: 0.820 vs 0.784. The 95% intervals exclude zero on 6 of 8 tasks and include it on 2.
  • CORE-BEHRT (0.802) ranks below both count baselines. We read a frozen [CLS] state where the original protocol fine-tunes the encoder, and it has no race or language inputs (App. A.1).

Logistic regression on counts was already present: the "TF-IDF bag" column in Table 3 is L2-regularized logistic regression on TF-IDF-weighted code counts, as §4.2 now states explicitly.

Retrieval baselines with trained models. Target-masked retrieval deliberately has no fitted model. It ranks patients by cosine similarity of frozen states, because the property under study is the geometry of the state: which patients it places near one another.

  • Supervised reference (App. A.3, Table 7). Following your suggestion, we trained per-phenotype logistic regression and GBT on training patients, using the same masked input that the retrieval arms see. We scored the test pool with the retrieval metric. They reach 0.912 and 0.934, against 0.701 for CHRONOS cosine. This measures how predictable each label is with supervision. Using no labels, CHRONOS covers 46% [46%, 47%] of GBT's margin over chance, and TF-IDF cosine covers 41%.
  • What we report. The comparison that bears on our claim is the within-model one. With the same architecture, data and state size, adding the fingerprint loss raises retrieval by +0.053 (+0.037 on INSPECT), and the gain survives PCA and record-footprint matching (Table 2).

Code and implementation checks. We agree that introducing a new model and benchmark alongside reimplemented baselines on MIMIC-IV requires careful verification. We recognize the value of independent code review and will release our code and run configurations upon acceptance.

We ran the following checks.

  • ORA. Our reimplementation uses the same data, byte-identical text embeddings, optimizer, and training budget. Its configuration differs from CHRONOS in four functional fields. The remaining differences are inactive schema fields (App. A.1).
  • CEHR-GPT runs through its upstream code and recipe. CORE-BEHRT is a clean re-implementation, with one-batch parity tested against upstream modules.
  • Masking. A masking verifier confirms that no deleted code reaches any input channel.
  • Significance inputs. We reproduced the stored AUROCs exactly from per-row scores. For TF-IDF, which lacked stored per-row scores, we refit the model using its saved settings and reproduced its AUROCs exactly.

We also identified and corrected three issues, which we describe in detail in our general response.

  • The recovery readout was missing an intercept. We corrected the implementation and updated the pooled metrics, while macro AUROC remained unchanged.
  • We clarified the retrieval masking procedure, which was incompletely described in the original submission.
  • We corrected our description of model selection to reflect that the slot count was chosen empirically. We also added an untuned replication using INSPECT to provide additional evidence for the findings.
Reviewer_9nAYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a new EHR foundation model that is trained with both prospective and retrospective objectives. It claims that a prospective pre-training objective is limited in retrieval and geometry-recovering tasks, and a retrospective loss function is also necessary.

Critical strengths
  1. This paper points out two important tasks that are often overlooked in previous EHR foundation models: retrospective accessibility and patient-similarity geometry.
  2. The paper proposes a novel loss function that consists of both prospective and retrospective terms. The well-designed ablation study shows that the retrospective loss function improves the overall downstream performance.
Critical weaknesses
  1. This paper has many confusing notations and equations with poor explanations:

(1) What is the difference between available time ai and clinical time ti?

(2) In line 152, how the window ω is sampled. Why would you exclude the window-only summary Φ(V_ωa)?

(3) What is readout family Gj (Line 160) and what is the meaning of operational decoding risk in equation 2?

  1. The comparison between CHRONOS and other baselines is not fair due to inconsistent model sizes. According to Appendix A.1, the parameter count of CHRONOS is approximately twice that of ORA and CORE-BEHRT. Also, CHRONOS has used Qwen3-embedding-8B to embed textual descriptions and different transformer architectures. It is hard to say whether the improvement comes from better input embeddings, the backbone, or the new loss function.

  2. The retrospective recovery tasks defined in the paper (line 297) are very similar to your loss function in section 3.5. They both have code occurrences within a certain window and the latest numeric values. It is hard to tell whether CHRONOS really outperforms other models in retrieval capabilities or simply overfits to a narrow range of tasks.

  3. CHRONOS only uses one dataset (MIMIC-IV). External validation on another dataset will strengthen the result.

  4. There is no code attached to the submission. It is hard to verify the correctness of implementations of your method and baselines (eg, ORA and CORE-BEHRT).

Questions for authors

Q1. How is multi-slot different from multi-head attention? Line 1138 says that “each slot is one attention head.”

Q2. How do you construct tasks cohorts for “Full history” vs “Omitted history” in table1. Are they classification tasks for a certain code or a regression task for a certain measurement? I also doubt the necessity of omitting history tasks, as most EHR trajectories are only a few hundred/thousand events. Using a larger context (eg, 8k) window should be enough for most patients.

Q3. Could you explain the retrieval task setup of target-masked patient retrieval in more detail? For each phenotype–medication in Table 4, how do you run the evaluation and compute the auroc? Also, how do you define the phenotype cohort? Do you construct cohorts on your own or use any previous benchmarks?

3 comments

Thank you for the constructive review. We answer the notation questions first, then fairness, the recovery tasks, external validation and your three questions.

(1) Clinical time tit_i vs availability time aia_i (now t~i\tilde t_i, to avoid a clash with the anchor index aa; §3.1, Table 17). tit_i is when a fact occurred, e.g. a lab's chart time; t~i=max⁡(ti,store time)\tilde t_i=\max(t_i,\text{store time}) is when it became readable in the EHR. The two differ for 75.4% of MIMIC events, almost all of them measurements (median store lag 70 minutes for labs, 2.6 days for microbiology). MIMIC diagnoses have no timestamp of their own and become available at discharge.

  • Inputs use t~i\tilde t_i: event order, blocks, cutoffs and the fingerprint history. The model therefore never sees a result before it was recorded.
  • Targets use tit_i, because "what happens next" is a clinical statement: prospective targets, the latest-value order and retrieval labels.
  • The rule matters. On INSPECT, without it 14% of PE-positive prediction points had a PE diagnosis dated in the preceding two days (1% of negatives); with it, 1.6% (0.3%).

(2) Window sampling, and why Φ(Vωa)\Phi(V_{\omega a}) is not the target (§3.1, App. F.1). Each epoch draws one window per patient: the whole record if it has at most 1,024 events, otherwise the longest complete-block suffix of at most 1,024 events, ending at a block end drawn uniformly among those that leave room for a full window. Up to 127 further block ends serve as prospective anchors, and the window end is always one. The fingerprint is computed at the window end from the full history HaH_a. "Excludes" was poor wording: we simply do not use the window-only summary as the target. The two coincide for the 54% of windows that hold the whole record; otherwise Φ(Ha)\Phi(H_a) also covers older events the encoder cannot see, which is what the omitted-history recovery results test.

(3) "Operational decoding risk." Rj(E)=inf⁡g∈GjE[ℓj(g(E(V)),Yj)]\mathcal R_j(E)=\inf_{g\in\mathcal G_j}\mathbb E[\ell_j(g(E(V)),Y_j)] is the lowest loss that a readout family Gj\mathcal G_j can achieve (j=+j=+: future events; j=−j=-: the history fingerprint). "Operational" means it is defined relative to that family and loss rather than information-theoretically. It is vocabulary only and never estimated, so we moved it to App. G.1. In training, G+\mathcal G_+ is the time-to-event head and G−\mathcal G_- the fingerprint head; evaluation uses other families (logistic probes, ridge readouts, cosine similarity).

Fairness: model size, Qwen embeddings, and architecture.

  • Parameters. The counts in App. A included pretraining heads that are discarded at export. App. A now also gives the exported encoders: 26.6M (CHRONOS), 23.2M (ORA), 11.8M (CORE-BEHRT) and 108.3M (CEHR-GPT). The largest, CEHR-GPT, ranks below both CHRONOS and ORA on prediction and retrieval.
  • Embeddings. Our ORA implementation uses the same frozen Qwen3-Embedding-8B table, data, optimizer and 10,000-step budget as CHRONOS. It differs only in the mask (causal vs block-causal), additive time features, a 1×512 last-event state instead of 16×128 slots, and the fingerprint.
  • Attribution. Training CHRONOS without the fingerprint splits the gap to ORA:
ORACHRONOS arch., no fingerprintCHRONOS
Prediction (16 tasks)0.8620.8690.871
Retrieval (8 phenotypes)0.6410.6480.701

From unrounded values, the architecture adds +0.007 on prediction and +0.008 on retrieval, and the fingerprint loss adds +0.002 and +0.053 with encoder, embeddings, data and state size fixed. The retrieval gain therefore comes from the loss. Comparisons with models that use no text embeddings (CEHR-GPT, CORE-BEHRT, TF-IDF, GBT) remain native-recipe context, as the paper states.

Recovery tasks resemble the loss. We agree; the recovery readouts are a manipulation check. They still differ from the loss: they are newly fitted ridge readouts rather than the trained head, they count diagnosis and DRG codes that the training count target excludes, and the omitted-history targets are events the encoder never receives.

Target-masked retrieval is the evidence that does not reuse the training targets:

  • The answer is not in the input. All task diagnosis and drug codes are deleted from the whole record before the window is chosen, and a verifier confirms that none reaches any input channel.
  • The loss never learns to fill in deleted codes. Training deletes nothing inside a window. When the window holds the whole record (54% of training windows, 49% of the retrieval pool), the target is the window's own codes, so a perfect fit would report a masked diagnosis as absent.
  • No trained readout. Retrieval is raw cosine between patient states; decodability by a head does not fix cosine geometry (App. G).
  • The label is temporal (a diagnosis followed by a drug order), whereas the fingerprint is order-free.

The gain survives PCA-32 and record-footprint matching (Table 2), costs nothing on the 16 prediction probes, and replicates on INSPECT (+0.037).

External validation (§5.5, App. D). We added INSPECT, a public OMOP dataset of 19,390 patients and 233M events from a different health system and coding system. Using its official split and the MIMIC recipe without retuning, we trained both arms of the matched pair from scratch.

  • Prediction. CHRONOS reaches 0.820 mean AUROC on the 8 benchmark tasks, versus 0.819 without the fingerprint and 0.784 for GBT on identical rows; CHRONOS − GBT excludes zero on all six mortality and readmission tasks.
  • Recovery. Codes occurring only before the input window: +0.023 macro AUROC (2,803 of 3,187 codes). Latest-value R2R^2: 0.181 → 0.323 (145 of 151 variables).
  • Retrieval. We rebuilt the 8 phenotype cohorts with the MIMIC rules (7 scorable, 17–94 queries each). Retrieval rises from 0.615 to 0.652, +0.037 [0.030, 0.043], on all 7 phenotypes, and the gain remains under PCA-32 and footprint matching (+0.021 to +0.034; Table 16).

Code. We will release the code and run configurations upon acceptance; App. A.1 documents how each baseline was trained and how ORA's configuration differs from ours.

Q1. Slots versus multi-head attention. Each slot is one 128-wide attention head with its own key and value projections and a learned query shifted by an anchor-time term. Unlike standard multi-head attention, the query is a parameter rather than a token projection, one 128×128 output map is shared by all slots instead of a dense 2,048×2,048 mix, and the 16 head outputs are exported side by side. There is no slot self-attention and no diversity loss. An earlier Perceiver-style design with shared keys and values collapsed (slot cosine similarity 1.000).

Q2. "Full" vs "omitted" history. One end-of-record snapshot per patient (111,638 train, 37,213 test). "Full" is every code up to the cutoff; "omitted" is codes that occur only before the ≤1,024-event window. Codes with at least 1% training prevalence get presence and log⁡(1+min⁡(n,32))\log(1+\min(n,32)) count targets (2,632 full, 1,754 omitted), plus 385 latest-value targets; one ridge readout is scored by macro AUROC and R2R^2 (App. B.3). You are right that 8,192 events would cover most patients: the median training record has 870 events and 7.8% exceed 8,192. Those long records, however, hold about 30% of all events, and at 1,024 events 46.2% of test patients are truncated, a limit every baseline shares. The omitted set isolates information the encoder cannot re-read, which is where retrospective and prospective objectives can differ.

Q3. Retrieval setup (§4.3, App. A.2, Table 5). We built the cohorts ourselves; each pairs a diagnosis with a later order for a drug that treats it, across specialties. The pool is the 18,568 test patients whose end-of-record window has some diagnosis followed by a prescription; a patient is positive if the original window has the phenotype's diagnosis followed by its drug, in clinical time. A fixed union of 55 diagnosis, prescription and administration codes is deleted from every pool record before the window is re-cut, and labels stay fixed (unmasked, the label codes alone reach AUROC 0.979). DRG codes naming a diagnosis stay in the input, and 575 patients who received the drug only through administration records count as negatives. Each positive ranks the other 18,567 patients by cosine; per-query AUROCs are averaged within, then across phenotypes, with nothing fitted. Two further cohorts, BPH → tamsulosin (age and sex alone reach 0.842) and depression → sertraline, are reported as a sensitivity analysis; the fingerprint improves both (+0.045 and +0.032; App. A.2).

Reviewer_nY1uOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the ability of a foundation model to retrospectively retrieve patient states in phenotype-retrieval cohorts, and how it can prospectively predict future clinical events simultaneously. They introduce CHRONOS, a patient-state model trained with both retrospective (fingerprint) and prospective (marked time-to-event) objectives. They also investigate representation geometry. The experiments are conducted using MIMIC-IV for 16 clinical prediction tasks.

Critical strengths
  • The paper introduces an interesting evaluation framework of foundation models, assessing (i) prospective predictiveness, (ii) retrospective accessibility, and (iii) patient-similarity geometry.
  • The results are obtained using a publicly available dataset.
Critical weaknesses
  • Some sentences are hard to understand like "A frozen state can be probed for whether a clinical code has occurred, how often it occurred, and what its latest value was"
  • The mathematical notation in Equation 1 is a bit difficult to parse and can be significantly simplified.
  • The set of baselines is quite limited as there is a vast number of EHR foundation models.
  • Additionally, can the framework be applied to other base foundation models?
  • Figure 1 presents results and it's shown too early.
  • There is no statistical significance testing.
  • It's unclear how the coefficients of the objective were fixed. The values are mentioned in the appendix only.
Questions for authors
  • Can you justify the choice of baselines?
  • "Architecture and hyperparameters were selected on validation data only" - what's the hyperparameter tuning strategy?
  • Will the code be made publicly available?
2 comments

Thank you for the detailed review. We respond to your comments below and provide an overview of the revisions in our general response.

W1. Hard-to-parse sentence. We replaced it with: "To measure retrospective accessibility, we freeze the patient state and fit a simple linear model that predicts, for each clinical code, whether the code appears in the patient's history, how many times it appears, and its most recent recorded value."

W2. Equation 1 notation. We simplified it to one line of set notation: Ha=d0∪{ei:t~i≤ua}H_a=d_0\cup\{e_i:\tilde t_i\le u_a\}, Vωa=d0∪{ei∈Wω:t~i≤ua}⊆HaV_{\omega a}=d_0\cup\{e_i\in W_\omega:\tilde t_i\le u_a\}\subseteq H_a, sωa=Eθ(Vωa)∈RKdss_{\omega a}=E_\theta(V_{\omega a})\in\mathbb R^{Kd_s}. In words: HaH_a is everything recorded by the cutoff, VωaV_{\omega a} is the part the encoder sees (at most 1,024 events), and Ha∖VωaH_a\setminus V_{\omega a} is older history it cannot see. We also removed two symbol clashes. Availability time is now t~i\tilde t_i, so it no longer clashes with the anchor index aa, and the admissible-cell set is now Yac\mathcal Y_{ac}. Finally, we removed the anchor-set and slot-matrix symbols, which the main text no longer needs.

W3 / Q1. Choice of baselines. We chose models that span the main families of EHR pretraining objectives and that we could retrain on the identical split and availability cutoffs (§4.2). Published checkpoints were trained on other data and splits, which would leave no clean held-out test set.

  • ORA is the prospective time-to-event objective that CHRONOS extends, in our re-implementation of the published method. It is the closest counterpart without retrospective supervision. It uses the same data, the same frozen text-embedding table (byte-identical), the same optimizer and the same 10,000-step budget, and it differs only in its mask, time features, state shape and the fingerprint.
  • CEHR-GPT (causal next-token) and CORE-BEHRT (bidirectional masked-event) cover the other two dominant objective families.
  • Count-based models. Logistic regression on TF-IDF counts was already in the paper; following Reviewer jy32, we added gradient-boosted trees (LightGBM) on code counts. GBT reaches 0.832 mean AUROC, versus 0.871 for CHRONOS. CHRONOS is significantly better on 14 of 16 tasks, and GBT is better on ICU length of stay.

We agree that many more EHR foundation models exist. Our central claim, however, is a within-model contrast: adding a retrospective objective improves accessibility and changes geometry without reducing prospective performance. The matched ablation, now on the final model and on a second dataset, tests that claim directly. The cross-model tables provide context.

W4. Can the framework be applied to other foundation models? Yes, in both senses.

  • The evaluation needs only a frozen state: linear probes for prediction, ridge readouts for recovery, and cosine retrieval. We already apply prediction and retrieval to all four baselines.
  • The fingerprint objective is a head on the exported state and does not depend on the encoder. We have trained it on two datasets (MIMIC-IV and INSPECT) and also attached it to a single-vector backbone: ORA's one 512-dimensional last-event state with the block-causal mask (App. C.3, Table 13). Retrieval improves by +0.043 [0.041, 0.046] on all 8 phenotypes, 82% of the gain in CHRONOS, and recovery of history before the input window improves by +0.008. That pair also differs in additive time features, so it is not an exact intervention.

W5. Figure 1 shows results too early. Figure 1 is now the model overview. The results figure moved to App. B (Figure 2), and its numbers are in Tables 1 and 2.

W6. Statistical significance. We added paired tests for both main tables (Tables 3, 4 and 6). Each comparison bootstraps the AUROC difference on identical test rows (2,000 replicates) and applies Benjamini–Hochberg correction across tasks.

CHRONOS minusPrediction, 16 tasks: Δ [95% CI]sig. for / againstRetrieval, 8 phenotypes: Δ [95% CI]sig. for / against
ORA+0.0087 [0.0047, 0.0129]5 / 0+0.0606 [0.0586, 0.0626]8 / 0
CEHR-GPT+0.0261 [0.0211, 0.0311]14 / 0+0.0950 [0.0921, 0.0982]8 / 0
CORE-BEHRT+0.0688 [0.0618, 0.0762]15 / 0+0.0904 [0.0866, 0.0944]8 / 0
GBT+0.0389 [0.0330, 0.0450]14 / 1——
TF-IDF+0.0541 [0.0474, 0.0609]15 / 0+0.0234 [0.0205, 0.0263]6 / 0

Retrieval intervals resample queries and hold the candidate pool fixed, so they are conditional on that pool.

The fingerprint ablations already reported bootstrap intervals; all of them exclude zero (Tables 2 and 9).

W7. How the loss coefficients were fixed. This is now in the main text (§3.6), with details in App. A.

  • The weights (λfp=0.3\lambda_{fp}=0.3, λcount=0.02\lambda_{count}=0.02, λvalue=0.05\lambda_{value}=0.05) were fixed before any fingerprint model was evaluated downstream. They used only training-batch gradients and a 640-patient validation subset, and no downstream labels or test data.
  • Component weights. We ran a 3×2 grid of 600-step runs on a smaller development variant and applied a rule fixed in advance. The rule takes the smallest cell in which the count and latest-value gradients on the shared encoder are each 0.5–2× the presence gradient. Validation prospective loss was flat across the grid. CHRONOS reuses these weights unchanged.
  • Outer weight. It comes from an earlier sweep of a code-presence (bag-of-words) auxiliary loss over {0.1, 1, 10}. A weight of 1 already made the auxiliary gradient ≈3× the prospective gradient and worsened prospective NLL. We set 0.3, between the tested values 0.1 and 1, and 0.3 itself was never a sweep point.
  • We also fixed a contradictory sentence in App. F (App. E in the submission).

Q2. Hyperparameter tuning strategy. We have corrected our original description of the model selection procedure (§5.6, App. A).

Validation loss was used to select the pretraining checkpoint, which was the final training step for all CHRONOS and ORA runs. Validation data were also used to select the prediction readout settings, including the regularization penalty, the number of neighbors kk, and the GBT configuration. Recovery readouts used either a training hold-out or a fixed regularization penalty. Test labels were never used to fit or select readouts.

We selected the number of slots empirically: we chose K=16K=16 because increasing to 32 slots added 8.4 million parameters without improving retrieval performance (§5.6). The optimizer settings followed standard defaults and were not tuned.

We also provide three results supporting the robustness of our findings to this choice. First, the matched ablation uses identical architectures and training settings, with the fingerprint objective as the only difference. Second, adding the fingerprint objective to a single-vector backbone improves retrieval AUROC by 0.043. Third, the INSPECT replication uses the final training configuration without additional tuning and reproduces the improvements in historical recovery and patient retrieval.

Q3. Code. Yes. We will release the code and run configurations upon acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision and new results

We thank the reviewers for their constructive feedback. Reviewers valued the three-axis view of a reusable patient state (nY1u, 9nAY), the matched ablation (9nAY), and the frozen-state AUROC protocol (jy32). The main concerns were clarity, the choice and strength of the baselines, significance testing, external validation, how hyperparameters were set, and code availability. The updated manuscript marks substantive changes in blue.

New results

  1. Paired significance tests for both main tables (Tables 3, 4, 6). We use 2,000 paired bootstrap replicates on the same test examples and apply Benjamini–Hochberg correction across tasks. On future prediction, CHRONOS outperforms ORA by +0.0087 AUROC (95% CI [0.0047, 0.0129]). The improvement is significant on 5 of 16 tasks, with no significant decreases. Against the other baselines, CHRONOS achieves significant improvements on 14–15 of 16 tasks. On retrieval, CHRONOS significantly outperforms each neural baseline on all eight phenotypes and TF-IDF on six.
  2. Gradient-boosted trees (LightGBM on code counts from the same window, tuned on validation; Table 3). Mean AUROC 0.832, versus 0.871 for CHRONOS. GBT is the best model on ICU length of stay above seven days.
  3. Matched fingerprint ablation of CHRONOS (Tables 1, 2). The arms differ only in the fingerprint flag. Raw retrieval improves by +0.053 (8 of 8 phenotypes) and recovery of history the encoder never sees by +0.012 macro AUROC, with no loss in prediction (+0.002). This also splits the CHRONOS–ORA gap: of +0.061 on retrieval, +0.053 comes from the objective and +0.008 from the architecture; on prediction, +0.002 and +0.007. On ORA's single-vector backbone the loss also raises retrieval, by +0.043 (Table 13).
  4. External replication on INSPECT (§5.5, App. D). INSPECT comes from a different health system and coding system. Using the MIMIC recipe without retuning, CHRONOS reaches 0.820 mean AUROC on the 8 benchmark tasks, versus 0.784 for GBT on identical rows. The fingerprint again improves recovery of history before the input window by +0.023 and raises target-masked retrieval by +0.037 on all 7 scored phenotypes (Table 16), without lowering prediction.
  5. Supervised retrieval reference (Table 7). We trained a separate LightGBM model for each phenotype using the same masked input. This supervised reference achieves a mean AUROC of 0.934. CHRONOS uses cosine similarity without phenotype-specific training and captures 46% of LightGBM's improvement over chance, compared with 41% for TF-IDF.

We also corrected the implementation or manuscript accordingly:

  • Recovery readout intercept. The ridge readout used for retrospective recovery had no intercept. Per-code macro AUROC is essentially unaffected (≤0.0003), but pooled metrics were biased. The corrected full-history gains are +0.006 micro AUROC (was +0.011) and +0.002 Recall@50 (was +0.011). Count-R2R^2 gains are unchanged (+0.05), and all gains remain significant. The targets are byte-identical; only the readout changed (Table 9).
  • Model selection. We corrected our statement that architecture and hyperparameters were selected exclusively using validation data. Validation data determined checkpoint selection and prediction readout settings. The number of slots was chosen empirically: 32 slots added 8.4M parameters without improving retrieval. The INSPECT experiments use the final configuration without additional tuning (§5.6, App. A).
  • Retrieval masking. We clarified that the same fixed set of 55 diagnosis, prescription, and administration codes is removed from every patient in the retrieval pool (§4.3, App. A.2), and verified that the deleted codes cannot enter the model through another input channel.
  • Baseline inputs. We corrected the description of sequence limits for CEHR-GPT and CORE-BEHRT. Their 1,024-token limits count tokens instead of clinical events, and both models exclude some demographic tokens. The revised manuscript describes these differences explicitly (§4.2, App. A.1).
  • Loss weights. We reconciled the inconsistent descriptions of how the objective weights were chosen. The procedure is now explained in §3.6 and App. A.

Presentation and implementation verification We rewrote the Methods section to explain the model and its training objectives more directly. Equation 1 has been simplified, and we clarify the distinction between clinical time and availability time, the training-window sampling procedure, and the multi-slot representation. Figure 1 now shows the model architecture, with the original results figure moved to App. B. We also added an overview at the beginning of §5 explaining how each experiment supports the paper's main claims.

Code. We agree that independent code review is valuable, particularly for a new model and evaluation framework using MIMIC-IV. We will release the implementation, evaluation code, and run configurations upon acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.