- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
STEP-CTS predicts outcomes from irregular clinical TS with a language model reader that consumes a set of units instead of the raw series or a generated summary. A Temporal Evidence Verbalizer (TEV) turns the trajectories into rule units consisting of a time window feature and value predicate. learned from gradient-boosted tree inventories at several look-back windows. Units are annotated with clinical concepts by an LLM. Then a learnable Evidence Selector (ES) picks the top units, and a BioClinical ModernBERT reader predicts from the rendered units. The method is evaluated on PhysioNet 2012 & 2019, and MIMIC-III in-hospital mortality. Authors compare it with text-serialization and summary baselines, time-series models and XGBoost.
The paper reports AUPRC gains of 3.7–11.7 percentage points over the best text baseline. It also reports a small concept-support audit, a latency comparison, and a two-clinician study concluding that the evidence is interpretable.
Critical strengths
transparent, well-controlled evaluation. Includes a random-initialized reader control frozen reader concept, deletion tests, matched budget sweeps, per-rater clinical tables, annotation controls and a sound limitation sections. I am confident in the numbers that are reported.
Critical weaknesses
The headline gain is driven by outcome-supervised rule discovery, not the language interface. Ablations show that a tuned XGBoost without text outperforms STEP-CTS, and text baselines lack the same supervised feature selection. The reported gains therefore compare a supervised interface against an unsupervised one. The do not isolate the benefit of the language interface itself.
The evidence for interpretability is weak and inconsistent. The paper claims the method is "above," "as interpretable as," and "comparable" to baselines. But two clinicians disagreed on its performance (near-zero inter-rater agreement), and LLM judges also did not show consistent preferable outcomes. Also, a concept audit showed that 41.5% of retained concepts were weak or overstated, but the headline reported the 4.5% that were unsupported. The study supports "no clear difference" in my perspective.
Questions for authors
Baselines: Can text baselines use the same outcome-supervised selection as the selector, or can the reader be evaluated on XGBoost-chosen units? What remains of the performance gains?
Input Length: Appendix B1 states max length is 3072 tokens, but Table K1 reports averages of 2,350 words. Is that aligned?
Clinician Study: Was the analysis pre-specified, raters blinded and independent, and IRB approved?
Variance: Can you repeat with different TEV fits or splits?
Audit: How is "concept support" defined, and what are the numbers if "weak/overstated" is counted as unsupported?
Cohort Counts: Please confirm the MIMIC-III cohort counts.
5 comments
Response to Reviewer g1uV (Part 2/3)
On the tuned XGBoost of Table 3 (XGB-tuned): it sums 100–200 rules per patient into a score with no clinical interpretation, and it does not outperform STEP-CTS overall. Each is higher on three of the six metrics, and STEP-CTS has the higher AUPRC on MIMIC-III and P2019.
Section 5 of the revision now attributes the gain over text baselines to the two parts that control (c) separates: the supervised rules and selection, and the text shown to the reader (threshold predicates with clinical concepts). It also reports control (b) in Table 2 (panel A): used alone as predictors, the per-window XGBoost models behind our rules are far weaker than XGB-tuned (Appendix C), but with the selector and LM reader trained on top, their rules give the better result on every dataset.
Weakness 2, Questions 3 and 5 (evidence for interpretability).
Clinician study (Weakness 2). The low item-level agreement comes mostly from a difference in level: clinician R2 scored on average about 0.9 points higher than clinician R1 on interpretation and usefulness. On which method is better, the two clinicians agree against Decode [1] and SHAP-to-Text [4] (same direction for 36 of 39 and 22 of 25 patients on interpretation) and disagree on STEP-CTS versus Record2Vec [3] (6 of 23; Table F1(c)). On these two scales, the two LLM judges also split only on STEP-CTS versus Record2Vec (Table F1(d)).
So the raters split on one comparison only, STEP-CTS versus Record2Vec, and there we agree that the study shows no clear difference. The rest is consistent, and it shows that each baseline does one thing while STEP-CTS does both. Decode [1] is as traceable as STEP-CTS, and both Decode and SHAP-to-Text [4] are rated lower on interpretation and usefulness. Record2Vec [3] is comparable on interpretation but far lower on traceability. STEP-CTS is the only presentation that both clinicians and both LLM judges place in the top two on every scale (Table F1). In the revision we replaced the inconsistent wording in the abstract, Contribution 3, Section 5, and the conclusion with "comparable to generated summaries on interpretation".
Study protocol (Question 3). As stated in Appendix F, the clinicians were blinded to the method and to the patient outcome. They rated independently of each other, and the rubric and the 50 patients were fixed before rating. The analysis was not pre-specified, so we report it as descriptive. The study used only P2012, a de-identified dataset that PhysioNet releases as open access. The clinicians took part as expert evaluators of model outputs, and no personal data about them were collected. The study therefore did not require IRB review. The revision states the protocol more clearly in Appendix F.
Concept audit (Weakness 2 and Question 5). We agree that the main text should not report the 4.5% alone, and Section 5 of the revision reports the full breakdown. A concept counts as supported when the audit grades it as entailed, that is, when the cited predicates guarantee the state for the whole window (Appendices E.2 and M.2). Of the 9,667 concepts, 53.9% are entailed, 41.5% weak, and 4.5% unsupported; counting weak as unsupported gives 46%. Weak means plausible but not guaranteed, or overstated relative to the cited predicates. For example, "maximum heart rate above 114" says only that the highest reading exceeded 114; the other readings in the window may be lower, so "tachycardia" is plausible but not guaranteed for the whole window. The predictive gain does not depend on these concepts: training with only the entailed ones preserves the AUPRC of the full model on every dataset (Table R1 in our response to Reviewer qYme).
Response to Reviewer g1uV (Part 3/3)
Question 2 (input length). Thank you for catching this; the 2,350 in Table K1 was our error. The reader input averages 893 words (1,884 tokens) per patient and is at most 2,106 tokens for the 200 patients of Table K1, below the 3,072-token limit. The revision has the corrected Table K1 below.
Table K1 (corrected). Display length for the 200 LLM-judged P2012 test patients.
| Statistic | Before grouping (reader input) | Grouped + rules without concepts |
|---|---|---|
| Words, mean (median) | 893 (896) | 647 (669) |
| Words, range | 653–990 | 185–1,088 |
| Tokens, mean (median) | 1,884 (1,886) | 1,631 (1,629) |
| Tokens, range | 1,485–2,106 | 564–2,437 |
Question 4 (variance across TEV fits). We refit the TEV on P2012 with three new XGBoost seeds, keeping the splits that all baselines share, and retrained the selector and the reader each time (Table R4 below; Table B7 in the new Appendix B.3). Performance is stable: AUPRC is 0.5181 ± 0.0049 over the three fits, against 0.5274 ± 0.0049 for the rule set used in the paper, and every refit is above the best text baseline (0.4768) and XGB-30 (0.5002).
Table R4. P2012; one training seed per refit, three for the rule set of the paper.
| XGBoost seed | Unique rules | Rules with a concept | Unique concepts | AUROC | AUPRC |
|---|---|---|---|---|---|
| 0 | 5,156 | 1,902 (36.9%) | 87 | 0.8527 | 0.5112 |
| 1 | 5,226 | 1,991 (38.1%) | 84 | 0.8526 | 0.5226 |
| 2 | 5,208 | 1,975 (37.9%) | 92 | 0.8568 | 0.5204 |
| Mean over fits | 5,197 ± 30 | 1,956 ± 39 (37.6 ± 0.5%) | 88 ± 3 | 0.8540 ± 0.0020 | 0.5181 ± 0.0049 |
| Paper rule set (seed 42) | 5,173 | 1,999 (38.6%) | 75 | 0.8561 ± 0.0017 | 0.5274 ± 0.0049 |
Question 6 (MIMIC-III cohort). The counts in Table A1(b) are correct, and all methods share this cohort. Table A1(c) was missing one exclusion step: the MIMIC-III reader of Horn et al. [2], which we use for preprocessing, has a built-in blacklist of 32 stays. The table below recounts every stage from the files, and the final counts match Table A1(b). The same check on P2012 found 12 records without a time series that the P2012 reader of Horn et al. [2] removes (train 9, val 3). The revision adds both blacklists to Table A1(c) and corrects the total number of excluded patients from 1,585 to 1,629.
| Stage | Train | Val | Test |
|---|---|---|---|
| Benchmark list files | 14,681 | 3,222 | 3,236 |
| Reader blacklist (> 1,000 distinct time points) | −19 | −5 | −8 |
| Sparsity exclusion | −30 | −9 | −11 |
| Final | 14,632 | 3,208 | 3,217 |
We hope these changes address your concerns. If you find them resolved, we would be grateful if you would consider updating your score.
[1] Fadlon, D., Dov, D., Bennett, A., Heller-Miron, D., Levy, G., Bar, K., and Weiss-Meilik, A. Decode Like a Clinician: Enhancing LLM Fine-Tuning with Temporal Structured Data Representation. IJCNLP-AACL, 2025.
[2] Horn, M., Moor, M., Bock, C., Rieck, B., and Borgwardt, K. Set Functions for Time Series. ICML, 2020.
[3] Ji, Z., Sun, Y., Amaral, A., Goldenberg, A., and Krishnan, R. G. Can We Generate Portable Representations for Clinical Time Series Data Using LLMs? ICLR, 2026.
[4] Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., and Lee, S.-I. From Local Explanations to Global Understanding with Explainable AI for Trees. Nature Machine Intelligence, 2020.
Response to Reviewer g1uV (Part 1/3)
We thank you for the careful reading, and especially for Question 1. XGB-tuned is a strong predictor, so we expected that integrating it with our selector and LM reader, in place of our own rules, would do at least as well as STEP-CTS. It does worse on every metric (Table R3). Before this control, the paper had not tested whether building the rules separately for each window size, as our TEV does, matters. This result shows that it does, and it is now a row of Table 2. The revision also adds the other controls and a TEV refit to the appendix, corrects Table K1 and Table A1(c), and revises the claims about interpretability. All changes in the revised PDF are marked in blue. We address each point below.
Table R3 (Table C3 in the revision). Mean ± s.d. over 3 seeds. Units: chosen by our selector. Rules: one XGBoost leaf path per tree.
Weakness 1 and Question 1 (outcome-supervised rules). We agree that our rules and selection are learned from outcome labels, while the text baselines of Table 1 build their text without them. We therefore ran the two controls asked for in Question 1 (Table R3 above): a text baseline given our supervised rules and selection (c), and the LM reader on XGBoost-chosen rules (a, b). With the same supervised rules and selection, the text baseline (c) is above the best text baseline and below STEP-CTS in AUPRC on every dataset (by 2.7 / 2.5 / 1.5 points on P2012 / MIMIC-III / P2019). So the supervision explains part of the gain, and the text shown to the reader explains the rest. Replacing the LM reader with an MLP over rule IDs also lowers AUPRC on every dataset (Table 2).