- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces CHRONOS, an EHR foundation model pretraining strategy that additionally pretrains by learning to predict prior state. The new pretraining objective does not improve risk prediction but does improve the ability to predict prior state.
Critical strengths
The primary evaluation metric for prospective risk, AUROC on a variety of tasks (as shown in Table 3), is correct, and the overall experimental setup seems roughly correct (although I cannot audit the code, so I can't speak with much confidence here).
Critical weaknesses
-
This is a very poorly written paper that is difficult to understand. It almost feels entirely AI written. For example, most of the subsections in results do not make any sense or appear to contribute to the claims at all. I don't understand what table 1 or table 2 have to do with the central claims of the paper. This paper needs to be rewritten from scratch by a human to improve clarity.
-
The baselines don't really make much sense. At minimum, both the prognostic and retrieval experiments should use gradient-boosted tree models. It looks like the retrieval baselines don't include any trained models? Not even a trained logistic regression model?
Questions for authors
Can you release your code so that it can be peer reviewed? This is a paper that introduces a novel algorithm, and then tests it on a new benchmark, comparing to baselines that are also all written by the authors. And the dataset involved here, MIMIC-IV, is very complex. There are so many opportunities for mistakes here, from top to bottom. Due diligence means a code review would be very useful here.
2 comments
Retrieval baselines with trained models. Target-masked retrieval deliberately has no fitted model. It ranks patients by cosine similarity of frozen states, because the property under study is the geometry of the state: which patients it places near one another.
- Supervised reference (App. A.3, Table 7). Following your suggestion, we trained per-phenotype logistic regression and GBT on training patients, using the same masked input that the retrieval arms see. We scored the test pool with the retrieval metric. They reach 0.912 and 0.934, against 0.701 for CHRONOS cosine. This measures how predictable each label is with supervision. Using no labels, CHRONOS covers 46% [46%, 47%] of GBT's margin over chance, and TF-IDF cosine covers 41%.
- What we report. The comparison that bears on our claim is the within-model one. With the same architecture, data and state size, adding the fingerprint loss raises retrieval by +0.053 (+0.037 on INSPECT), and the gain survives PCA and record-footprint matching (Table 2).
Code and implementation checks. We agree that introducing a new model and benchmark alongside reimplemented baselines on MIMIC-IV requires careful verification. We recognize the value of independent code review and will release our code and run configurations upon acceptance.
We ran the following checks.
- ORA. Our reimplementation uses the same data, byte-identical text embeddings, optimizer, and training budget. Its configuration differs from CHRONOS in four functional fields. The remaining differences are inactive schema fields (App. A.1).
- CEHR-GPT runs through its upstream code and recipe. CORE-BEHRT is a clean re-implementation, with one-batch parity tested against upstream modules.
- Masking. A masking verifier confirms that no deleted code reaches any input channel.
- Significance inputs. We reproduced the stored AUROCs exactly from per-row scores. For TF-IDF, which lacked stored per-row scores, we refit the model using its saved settings and reproduced its AUROCs exactly.
We also identified and corrected three issues, which we describe in detail in our general response.
- The recovery readout was missing an intercept. We corrected the implementation and updated the pooled metrics, while macro AUROC remained unchanged.
- We clarified the retrieval masking procedure, which was incompletely described in the original submission.
- We corrected our description of model selection to reflect that the slot count was chosen empirically. We also added an untuned replication using INSPECT to provide additional evidence for the findings.
Thank you for the comments. We agree that the submission was hard to follow, and we have revised it substantially. We also agree with your point about baselines and have added gradient-boosted trees.
Clarity, and what Tables 1 and 2 have to do with the claims. The central claim is narrow: a patient state's prospective performance does not determine how much of its past history is accessible or which patients are its neighbors. Each table now has a stated role (§5, new overview paragraph):
Concrete changes:
Baselines: gradient-boosted trees for prediction. We added LightGBM on raw code counts from the same input window and rows. It is tuned on validation (learning rate × leaves grid, early stopping) and scored once on test.
Logistic regression on counts was already present: the "TF-IDF bag" column in Table 3 is L2-regularized logistic regression on TF-IDF-weighted code counts, as §4.2 now states explicitly.