acceptodds
Under review as a conference paper at ICLR 2027

Learning to Select Source-Traceable Evidence for Language-Model Prediction from Irregular Clinical Time Series

Abstract

Numerical time-series models effectively process irregular electronic health record (EHR) trajectories, but do not expose which temporal patterns support each prediction as readable evidence. Existing text-based interfaces either serialize observations, preserving source traceability but offering limited clinical interpretation, or generate patient-level summaries that improve readability but can obscure links to source measurements. We introduce STEP-CTS (Source-Traceable Evidence for Prediction from Clinical Time-Series), which learns to select source-traceable text evidence for a language-model predictor. Multi-scale window statistics of each trajectory are verbalized as sets of deterministic threshold predicates, each set linked to its source window (e.g., "[5–7 h]: last temperature at least 38C"). An offline LLM, run once per unique predicate set without access to patient records, the prediction task, or outcome labels, attaches clinical concepts such as "fever" to their supporting predicates and abstains when none applies. Each predicate set, together with its concepts, forms an evidence unit. A learned Evidence Selector selects a fixed-size subset of the evidence units, which a pretrained clinical language-model encoder reads to make the prediction, with no patient-level text generation. Across three ICU benchmarks, STEP-CTS outperforms evaluated text-based baselines, improving AUPRC over the strongest by 5.1, 3.7, and 11.7 percentage points on P2012, MIMIC-III, and P2019, respectively, and is competitive with dedicated numerical time-series models. Ablations show that clinical concepts and learned evidence selection each contribute to predictive performance. In a blinded clinician study, the selected evidence is rated as traceable as deterministic serialization, near the ceiling of the scale, and above generated summaries on clinical interpretation.

Primary area:
interpretability and explainable AI
Keywords:
clinical time series, clinical language models, interpretable machine learning, irregular time series, clinical prediction, evidence selection

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_g1uVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

STEP-CTS predicts outcomes from irregular clinical TS with a language model reader that consumes a set of units instead of the raw series or a generated summary. A Temporal Evidence Verbalizer (TEV) turns the trajectories into rule units consisting of a time window feature and value predicate. learned from gradient-boosted tree inventories at several look-back windows. Units are annotated with clinical concepts by an LLM. Then a learnable Evidence Selector (ES) picks the top units, and a BioClinical ModernBERT reader predicts from the rendered units. The method is evaluated on PhysioNet 2012 & 2019, and MIMIC-III in-hospital mortality. Authors compare it with text-serialization and summary baselines, time-series models and XGBoost.

The paper reports AUPRC gains of 3.7–11.7 percentage points over the best text baseline. It also reports a small concept-support audit, a latency comparison, and a two-clinician study concluding that the evidence is interpretable.

Critical strengths

transparent, well-controlled evaluation. Includes a random-initialized reader control frozen reader concept, deletion tests, matched budget sweeps, per-rater clinical tables, annotation controls and a sound limitation sections. I am confident in the numbers that are reported.

Critical weaknesses

The headline gain is driven by outcome-supervised rule discovery, not the language interface. Ablations show that a tuned XGBoost without text outperforms STEP-CTS, and text baselines lack the same supervised feature selection. The reported gains therefore compare a supervised interface against an unsupervised one. The do not isolate the benefit of the language interface itself.

The evidence for interpretability is weak and inconsistent. The paper claims the method is "above," "as interpretable as," and "comparable" to baselines. But two clinicians disagreed on its performance (near-zero inter-rater agreement), and LLM judges also did not show consistent preferable outcomes. Also, a concept audit showed that 41.5% of retained concepts were weak or overstated, but the headline reported the 4.5% that were unsupported. The study supports "no clear difference" in my perspective.

Questions for authors

Baselines: Can text baselines use the same outcome-supervised selection as the selector, or can the reader be evaluated on XGBoost-chosen units? What remains of the performance gains?

Input Length: Appendix B1 states max length is 3072 tokens, but Table K1 reports averages of 2,350 words. Is that aligned?

Clinician Study: Was the analysis pre-specified, raters blinded and independent, and IRB approved?

Variance: Can you repeat with different TEV fits or splits?

Audit: How is "concept support" defined, and what are the numbers if "weak/overstated" is counted as unsupported?

Cohort Counts: Please confirm the MIMIC-III cohort counts.

5 comments
AuthorsOpenReview ↗

Response to Reviewer g1uV (Part 1/3)

We thank you for the careful reading, and especially for Question 1. XGB-tuned is a strong predictor, so we expected that integrating it with our selector and LM reader, in place of our own rules, would do at least as well as STEP-CTS. It does worse on every metric (Table R3). Before this control, the paper had not tested whether building the rules separately for each window size, as our TEV does, matters. This result shows that it does, and it is now a row of Table 2. The revision also adds the other controls and a TEV refit to the appendix, corrects Table K1 and Table A1(c), and revises the claims about interpretability. All changes in the revised PDF are marked in blue. We address each point below.

Table R3 (Table C3 in the revision). Mean ± s.d. over 3 seeds. Units: chosen by our selector. Rules: one XGBoost leaf path per tree.

SettingPer patientP2012 AUROCP2012 AUPRCMIMIC-III AUROCMIMIC-III AUPRCP2019 AUROCP2019 AUPRC
STEP-CTS (Ours)30 units0.8561±0.00170.5274±0.00490.8563±0.00240.5025±0.00560.9115±0.00160.5004±0.0157
STEP-CTS without concepts (Table 2)30 units0.8510±0.00010.4997±0.00430.8503±0.00150.4650±0.01580.9147±0.00480.4783±0.0040
XGB-30 (Table 3)30 rules0.8482±0.00220.5002±0.00460.8391±0.00050.4290±0.01060.9051±0.00110.4208±0.0006
XGB-tuned (Table 3)200 / 100 / 100 rules0.8716±0.00050.5371±0.00180.8552±0.00060.4897±0.01250.9279±0.00020.4802±0.0034
Decode-Adapted (best text baseline, Table 1)all observations0.8444±0.00310.4768±0.00010.8474±0.00260.4657±0.00600.8943±0.00410.3836±0.0011
(a) Rules of XGB-30 → LM reader, predicates only30 rules0.8320±0.00320.4727±0.01120.8402±0.00170.4532±0.00000.9146±0.00010.4512±0.0029
(a) Rules of XGB-30 → LM reader, predicates with concepts30 rules0.8371±0.00430.4796±0.01650.8435±0.00090.4623±0.00060.9136±0.00370.4689±0.0045
(b) Rules of XGB-tuned → selector (K=30) → LM reader, predicates with concepts30 units0.8384±0.00070.4908±0.00450.8432±0.00080.4239±0.00750.9078±0.00390.4513±0.0041
(c) Units of our selector in Decode format → LM reader30 units0.8407±0.00580.5005±0.01650.8518±0.00090.4778±0.00410.9128±0.00130.4859±0.0031

Weakness 1 and Question 1 (outcome-supervised rules). We agree that our rules and selection are learned from outcome labels, while the text baselines of Table 1 build their text without them. We therefore ran the two controls asked for in Question 1 (Table R3 above): a text baseline given our supervised rules and selection (c), and the LM reader on XGBoost-chosen rules (a, b). With the same supervised rules and selection, the text baseline (c) is above the best text baseline and below STEP-CTS in AUPRC on every dataset (by 2.7 / 2.5 / 1.5 points on P2012 / MIMIC-III / P2019). So the supervision explains part of the gain, and the text shown to the reader explains the rest. Replacing the LM reader with an MLP over rule IDs also lowers AUPRC on every dataset (Table 2).

  • Same supervised rules and selection, different text (c). We kept our outcome-supervised rules and selection and changed only the text: each selected unit is written as measured values in the Decode [1] format, without thresholds or concepts. In AUPRC, this is higher than Decode-Adapted, the best text baseline, on every dataset, so the supervised selection does explain part of our gain; but it is still lower than STEP-CTS on every dataset, so with the supervision held the same, the text (threshold predicates with clinical concepts) still adds to the result.
  • Same supervised rules and selection, no text. With only the LM reader replaced by an MLP over rule IDs, AUPRC drops on every dataset (Table 2).
  • XGBoost-chosen rules, same interface (a, b). In (a), we trained the same LM reader on the 30 rules that XGB-30 (Table 3) uses for each patient, one per tree. In (b), we took the rules of XGB-tuned as candidates and trained our selector (K=30) and the LM reader on them from the start, as in STEP-CTS. Both have concepts attached; without them (a) is only slightly lower, so the gap lies in the rules, not in the concepts. Both are below STEP-CTS in AUPRC on every dataset, and (b) is below (a) on MIMIC-III and P2019 although XGB-tuned is a stronger predictor than XGB-30. So outcome-supervised rules alone do not explain our result either. The two kinds of rules differ in how they are built: a rule of XGB-tuned is one of 100–200 terms of a sum and can combine statistics from several window sizes, while a rule of our TEV is built for one window size and describes the patient's state in one time window.
AuthorsOpenReview ↗

Response to Reviewer g1uV (Part 2/3)

On the tuned XGBoost of Table 3 (XGB-tuned): it sums 100–200 rules per patient into a score with no clinical interpretation, and it does not outperform STEP-CTS overall. Each is higher on three of the six metrics, and STEP-CTS has the higher AUPRC on MIMIC-III and P2019.

Section 5 of the revision now attributes the gain over text baselines to the two parts that control (c) separates: the supervised rules and selection, and the text shown to the reader (threshold predicates with clinical concepts). It also reports control (b) in Table 2 (panel A): used alone as predictors, the per-window XGBoost models behind our rules are far weaker than XGB-tuned (Appendix C), but with the selector and LM reader trained on top, their rules give the better result on every dataset.

Weakness 2, Questions 3 and 5 (evidence for interpretability).

Clinician study (Weakness 2). The low item-level agreement comes mostly from a difference in level: clinician R2 scored on average about 0.9 points higher than clinician R1 on interpretation and usefulness. On which method is better, the two clinicians agree against Decode [1] and SHAP-to-Text [4] (same direction for 36 of 39 and 22 of 25 patients on interpretation) and disagree on STEP-CTS versus Record2Vec [3] (6 of 23; Table F1(c)). On these two scales, the two LLM judges also split only on STEP-CTS versus Record2Vec (Table F1(d)).

So the raters split on one comparison only, STEP-CTS versus Record2Vec, and there we agree that the study shows no clear difference. The rest is consistent, and it shows that each baseline does one thing while STEP-CTS does both. Decode [1] is as traceable as STEP-CTS, and both Decode and SHAP-to-Text [4] are rated lower on interpretation and usefulness. Record2Vec [3] is comparable on interpretation but far lower on traceability. STEP-CTS is the only presentation that both clinicians and both LLM judges place in the top two on every scale (Table F1). In the revision we replaced the inconsistent wording in the abstract, Contribution 3, Section 5, and the conclusion with "comparable to generated summaries on interpretation".

Study protocol (Question 3). As stated in Appendix F, the clinicians were blinded to the method and to the patient outcome. They rated independently of each other, and the rubric and the 50 patients were fixed before rating. The analysis was not pre-specified, so we report it as descriptive. The study used only P2012, a de-identified dataset that PhysioNet releases as open access. The clinicians took part as expert evaluators of model outputs, and no personal data about them were collected. The study therefore did not require IRB review. The revision states the protocol more clearly in Appendix F.

Concept audit (Weakness 2 and Question 5). We agree that the main text should not report the 4.5% alone, and Section 5 of the revision reports the full breakdown. A concept counts as supported when the audit grades it as entailed, that is, when the cited predicates guarantee the state for the whole window (Appendices E.2 and M.2). Of the 9,667 concepts, 53.9% are entailed, 41.5% weak, and 4.5% unsupported; counting weak as unsupported gives 46%. Weak means plausible but not guaranteed, or overstated relative to the cited predicates. For example, "maximum heart rate above 114" says only that the highest reading exceeded 114; the other readings in the window may be lower, so "tachycardia" is plausible but not guaranteed for the whole window. The predictive gain does not depend on these concepts: training with only the entailed ones preserves the AUPRC of the full model on every dataset (Table R1 in our response to Reviewer qYme).

AuthorsOpenReview ↗

Response to Reviewer g1uV (Part 3/3)

Question 2 (input length). Thank you for catching this; the 2,350 in Table K1 was our error. The reader input averages 893 words (1,884 tokens) per patient and is at most 2,106 tokens for the 200 patients of Table K1, below the 3,072-token limit. The revision has the corrected Table K1 below.

Table K1 (corrected). Display length for the 200 LLM-judged P2012 test patients.

StatisticBefore grouping (reader input)Grouped + rules without concepts
Words, mean (median)893 (896)647 (669)
Words, range653–990185–1,088
Tokens, mean (median)1,884 (1,886)1,631 (1,629)
Tokens, range1,485–2,106564–2,437

Question 4 (variance across TEV fits). We refit the TEV on P2012 with three new XGBoost seeds, keeping the splits that all baselines share, and retrained the selector and the reader each time (Table R4 below; Table B7 in the new Appendix B.3). Performance is stable: AUPRC is 0.5181 ± 0.0049 over the three fits, against 0.5274 ± 0.0049 for the rule set used in the paper, and every refit is above the best text baseline (0.4768) and XGB-30 (0.5002).

Table R4. P2012; one training seed per refit, three for the rule set of the paper.

XGBoost seedUnique rulesRules with a conceptUnique conceptsAUROCAUPRC
05,1561,902 (36.9%)870.85270.5112
15,2261,991 (38.1%)840.85260.5226
25,2081,975 (37.9%)920.85680.5204
Mean over fits5,197 ± 301,956 ± 39 (37.6 ± 0.5%)88 ± 30.8540 ± 0.00200.5181 ± 0.0049
Paper rule set (seed 42)5,1731,999 (38.6%)750.8561 ± 0.00170.5274 ± 0.0049

Question 6 (MIMIC-III cohort). The counts in Table A1(b) are correct, and all methods share this cohort. Table A1(c) was missing one exclusion step: the MIMIC-III reader of Horn et al. [2], which we use for preprocessing, has a built-in blacklist of 32 stays. The table below recounts every stage from the files, and the final counts match Table A1(b). The same check on P2012 found 12 records without a time series that the P2012 reader of Horn et al. [2] removes (train 9, val 3). The revision adds both blacklists to Table A1(c) and corrects the total number of excluded patients from 1,585 to 1,629.

StageTrainValTest
Benchmark list files14,6813,2223,236
Reader blacklist (> 1,000 distinct time points)−19−5−8
Sparsity exclusion−30−9−11
Final14,6323,2083,217

We hope these changes address your concerns. If you find them resolved, we would be grateful if you would consider updating your score.

[1] Fadlon, D., Dov, D., Bennett, A., Heller-Miron, D., Levy, G., Bar, K., and Weiss-Meilik, A. Decode Like a Clinician: Enhancing LLM Fine-Tuning with Temporal Structured Data Representation. IJCNLP-AACL, 2025.

[2] Horn, M., Moor, M., Bock, C., Rieck, B., and Borgwardt, K. Set Functions for Time Series. ICML, 2020.

[3] Ji, Z., Sun, Y., Amaral, A., Goldenberg, A., and Krishnan, R. G. Can We Generate Portable Representations for Clinical Time Series Data Using LLMs? ICLR, 2026.

[4] Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., and Lee, S.-I. From Local Explanations to Global Understanding with Explainable AI for Trees. Nature Machine Intelligence, 2020.

Reviewer_qYmeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

STEP-CTS constructs a bounded, source-linked evidence set from longitudinal clinical measurements and feeds it to a language model for prediction. A learned selector chooses rule-derived units, optionally augmented with clinical concept descriptions. Unlike a post-hoc explanation, the selected evidence is part of the predictor's actual input, making the information bottleneck itself inspectable.

Critical strengths
  • The source-linked bottleneck is the strongest part of the idea. It connects the displayed evidence to what the model actually consumes, instead of generating a separate explanation after prediction.

  • The predictive evaluation includes substantial controls: numerical models, evidence-budget comparisons, a rule-ID reader, language-model initialization variants, and concept ablations. Appendix E also varies concept retention while fixing the selected predicates and reader. These experiments provide meaningful evidence for the contributions of selection, reading, and concept augmentation.

Critical weaknesses
  • The main unresolved issue is the semantic step from numerical predicates to clinical concepts. The audit reports 53.9% of rule-concept pairs as entailed, 41.5% as weak or overstated, and 4.5% as unsupported or contradicted. A one-sided numerical bound may be compatible with a clinical state without establishing it. Source traceability therefore does not, by itself, make the concept description clinically faithful. This matters because these descriptions are both prediction features and part of the clinician-facing evidence.

  • The concept ablations show that the model benefits from the labels, but not that clinically justified labels are responsible for that benefit. Likewise, the clinician study compares different evidence-generation pipelines rather than isolating the presentation benefit of adding concepts to the same selected predicates. The predictive bottleneck is better established than the claim that its conceptual layer improves clinical interpretation. This distinction should be central to the conclusions, not treated as a minor annotation-quality issue.

Questions for authors
  1. What happens when weak/unsupported concepts are removed, qualified, or repaired while preserving the selected numerical predicates? Evaluating this on the evidence actually selected at inference would test whether predictive gains survive better semantic grounding; the existing concept-retention experiment does not separate label validity.

  2. For the same selected evidence, do clinicians prefer predicates alone, predicates with the original concepts, or predicates with qualified concepts? A paired comparison would isolate whether the conceptual layer improves interpretation without overstating what the measurements establish.

3 comments
AuthorsOpenReview ↗

Response to Reviewer qYme (Part 1/2)

We thank you for the careful reading and for stating precisely what the paper does and does not establish. Your comments led us to run three additional experiments, which are now in Appendices E.3 and F.2 of the revision (Tables R1, R2, and R6 below), and to revise the claims in the abstract and the conclusion. We believe this strengthens the paper. All changes in the revised PDF are marked in blue. We address each point below.

Weakness 1 and Question 1 (clinical faithfulness of the concepts). We agree that source traceability does not by itself make the attached concepts clinically faithful, and the revised conclusion now says so explicitly. This risk comes with delegating clinical interpretation to an LLM, which STEP-CTS shares with other LLM-based approaches like Record2Vec [1] and TimeCP [2]. What STEP-CTS adds is that the risk can be checked in advance. The LLM writes concepts once per rule, offline, and not per patient, so everything it can ever show is a fixed set of rule-concept pairs, and each pair can be checked once, against the predicates it cites, for all current and future patients. A per-patient summary cannot be checked this way: it would need a separate check for every patient.

We ran this check (Appendix E.2): 53.9% of the concepts are graded as entailed, 41.5% as weak, and 4.5% as unsupported. Since concepts are also inputs to the LM reader, the question is whether the predictive gain needs the 46% that are not entailed. It does not (Table R1 below; Table E3 in the new Appendix E.3 of the revision). Numbers are AUPRC on P2012 / MIMIC-III / P2019.

  • Training with only the entailed concepts (f) gives 0.5311 / 0.5028 / 0.5001, within one standard deviation of the full model (0.5274 / 0.5025 / 0.5004) and above the model trained without concepts (g) on every dataset. AUROC stays within 0.8 points of (a) and is lower on MIMIC-III.
  • Unsupported concepts alone (e) are no better than no concepts (d) on any dataset.
  • Removing concepts at test time, with the selector and reader frozen. Without the unsupported concepts (c), AUPRC changes by at most 0.3 points. Without the weak concepts as well (b), it drops by at most 1.0 point and stays above no concepts (d) on every dataset. This drop disappears in (f), where the reader is trained without them.

The audit can therefore be used as a filter. Of the options you list, we tested the strictest one, removal, and the gain is preserved. We keep the full model as the main result and report the entailed-only model in Appendix E.3.

Table R1. The 9,667 concepts are split by the audit grade of Appendix E.2: 5,215 entailed (grade 2), 4,016 weak (grade 1), 436 unsupported (grade 0). (b)–(e): concept text removed at test time, with the selector and reader of (a) frozen; the 30 selected units are the same as in (a). (f)–(g): selector and reader trained with the reduced set of concepts, under the same protocol as (a). Mean ± s.d. over 3 seeds.

Setting# conceptsP2012 AUROCP2012 AUPRCMIMIC-III AUROCMIMIC-III AUPRCP2019 AUROCP2019 AUPRC
(a) All concepts (Ours)9,6670.8561±0.00170.5274±0.00490.8563±0.00240.5025±0.00560.9115±0.00160.5004±0.0157
(b) Entailed only5,2150.8534±0.00100.5207±0.00070.8543±0.00350.4997±0.01170.9089±0.00330.4908±0.0116
(c) Entailed + weak9,2310.8553±0.00080.5248±0.00180.8560±0.00340.5016±0.01070.9115±0.00170.5000±0.0153
(d) No concepts00.8527±0.00010.5090±0.00130.8530±0.00140.4949±0.00230.9037±0.00520.4763±0.0093
(e) Unsupported only4360.8503±0.00000.5086±0.00070.8531±0.00360.4814±0.01470.9054±0.00370.4735±0.0088
(f) Trained with entailed concepts only5,2150.8595±0.00160.5311±0.00390.8487±0.00680.5028±0.01220.9123±0.00310.5001±0.0097
(g) Trained without concepts (Table 2)00.8510±0.00010.4997±0.00430.8503±0.00150.4650±0.01580.9147±0.00480.4783±0.0040
AuthorsOpenReview ↗

Response to Reviewer qYme (Part 2/2)

Weakness 2 and Question 2 (isolating the concept layer). We agree with this distinction. Table 4 compares whole pipelines and does not isolate the concept layer. To isolate it, we ran the paired comparison suggested in Question 2, with LLM judges and with clinicians.

Are clinically justified labels responsible for the benefit? Table R1 above tests this. Training with only the concepts graded as entailed preserves the gain (f), and the unsupported concepts alone give none (e), though they are only 4.5% of all concepts. Labeling more rules does not help either: removing abstention lowers AUPRC by 2.9 / 1.7 / 3.4 points (Appendix Table D1). In the revision, Section 5 reports the full audit breakdown (53.9% entailed, 41.5% weak, 4.5% unsupported), not the 4.5% alone, and the conclusion states that traceability does not make the attached concepts clinically faithful.

Paired comparison by LLM judges. As suggested, we took the same 200 P2012 test patients as in Table F1(d), and both LLM judges rated again the same selected units with and without concepts (Table R2 below; Table F2 in the new Appendix F.2, with an example in Appendix K.4). For both judges, concepts raise clinical usefulness and interpretation quality. Source traceability stays at its already high level, because the predicates alone are source-traceable.

Table R2. 200 P2012 test patients; mean ± s.d. over 400 ratings per cell (2 passes).

JudgeAxisPredicates onlyPredicates with concepts
Claude Sonnet 5Source traceability4.00 ± 0.124.07 ± 0.33
Claude Sonnet 5Clinical interpretation quality1.02 ± 0.142.80 ± 0.51
Claude Sonnet 5Clinical usefulness1.80 ± 0.403.02 ± 0.52
GPT-5.5Source traceability4.99 ± 0.124.87 ± 0.39
GPT-5.5Clinical interpretation quality1.00 ± 0.003.91 ± 0.62
GPT-5.5Clinical usefulness2.00 ± 0.053.96 ± 0.55

Paired comparison by clinicians. The two clinicians of Table 4 rated the same two renderings, in randomized order, for 20 P2012 test patients (Table R6 below; Table F3 in the new Appendix F.2). These 20 patients are not among the 50 of Table 4, so the clinicians had not seen them before. As with the LLM judges, concepts raise clinical usefulness and interpretation quality for both clinicians. The interpretation scale asks raters to lower the score for overstated claims (Appendix M.3), and it still rises with concepts. Source traceability changes little (4.58 to 4.40). Both commented that grouping by concept made it easier to see why the model made its prediction, while the predicates-only version lists summary measurements without naming the clinical problem (Appendix F.2). With 20 patients we report this as descriptive. Section 5 of the revision reports this paired comparison as the evidence for the concept layer.

Table R6. 20 P2012 test patients; mean ± s.d. over 40 ratings per cell (two clinicians). Per-clinician scores are in Table F3.

AxisPredicates onlyPredicates with concepts
Source traceability4.58 ± 0.504.40 ± 0.74
Clinical interpretation quality2.33 ± 0.773.35 ± 0.66
Clinical usefulness2.23 ± 0.664.35 ± 0.83

[1] Ji, Z., Sun, Y., Amaral, A., Goldenberg, A., and Krishnan, R. G. Can We Generate Portable Representations for Clinical Time Series Data Using LLMs? ICLR, 2026.

[2] Lee, G., Yu, W., Shin, K., Cheng, W., and Chen, H. TimeCAP: Learning to Contextualize, Augment, and Predict Time Series Events with Large Language Model Agents. AAAI, 2025.

Dear Reviewer qYme,

Thank you for taking the time to read and consider our response. Following your two questions, we added a model trained with only the entailed concepts (Table E3) and a paired comparison of the same evidence with and without concepts, rated by clinicians and LLM judges (Tables F2 and F3).

If any point remains open, we would be glad to answer it within the discussion period. Otherwise, we would be grateful if you would consider updating your score.

Reviewer_ujxEOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This study proposes STEP-CTS (Source-Traceable Evidence for Prediction from Clinical Time-Series), which learns to select source-traceable text evidence for a language-model predictor. They introduce an offline LLM, run once per unique predicate set without access to patient records, the prediction task, or outcome labels, attaches clinical concepts such as “fever” to their supporting predicates and abstains when none applies. And they proposes a learned Evidence Selector to select a fixed-size subset of the evidence units. Across three ICU benchmarks, they demonstrate STEP-CTS outperforms evaluated text-based baselines, improving AUPRC over the strongest by 5.1, 3.7, and 11.7 percentage points on P2012, MIMIC-III, and P2019, respectively, and is competitive with dedicated numerical time-series models.

Critical strengths

The research problem addressed in the paper is significant and holds practical value. Furthermore, the paper demonstrates a certain degree of innovation, and the experimental design is sound.

Critical weaknesses

It is recommended to address some concerns, including that the training objective for the Evidence Selector lacks sufficient justification; the sampling process consists of Bernoulli sampling followed by top-K truncation; consequently, the final selection probabilities do not follow a simple, independent Bernoulli distribution. In the loss function of equation (4), is an indicator variable derived from a top-K truncation; however, the corresponding gradient is not equivalent to the gradient of the true composite top-K policy.

Questions for authors

1 The main methodological contribution appears to combine multi-scale window statistics, XGBoost-derived threshold rules, LLM-generated clinical concepts, and a learned evidence selector. Could the authors clarify what is fundamentally new compared with existing EHR serialization, rationale-selection, and language-model-based clinical prediction methods?

2 The selector is trained with a self-critical policy-gradient objective, but the actual selection process consists of Bernoulli sampling followed by top-K truncation. Please explain this concern.

3 The clinical concepts are generated by Qwen3.5-27B and evaluated by Claude Sonnet 5. Could the performance gains simply reflect semantic information injected by a large language model, rather than the proposed evidence-selection mechanism?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ujxE (Part 1/2)

We thank you for the careful reading. Your comment on the selector objective led us to find and correct an error in how Eq. (4) was written and to run two further experiments on the selector loss, which we believe further strengthen the paper. All changes in the revised PDF are marked in blue. We address each point below.

Question 1 (what is new?). The individual components build on known techniques. What is new is the text they jointly produce for the LM reader: K evidence units chosen by a learned selector. Each unit is computed from the patient's irregular time series by fixed rules and is linked to its variables and time window, so every evidence unit the LM reader reads can be followed back to the measurements behind it. When a clinical concept applies, the unit also carries one. The LLM writes concepts once per rule, offline, and not per patient and not during test time, so its output is a finite set of rule-concept pairs, and each pair is checked against the predicates it cites, for all patients. A per-patient LLM summary cannot be checked this way; it would need a separate faithfulness check for every patient one by one.

None of the following three lines of work gives the reader text that is source-traceable, clinically labeled, and selected at the same time.

  • EHR serialization (e.g., Decode [1]) is deterministic and traceable, but it verbalizes every observation, adds no clinical interpretation, and does not learn what to show.
  • Rationale selection [2, 5] selects tokens from a text that already exists. An irregular time series has no such text, so STEP-CTS first constructs the candidates, each tied to its source window and rule, and then selects among them.
  • LM-based clinical prediction uses an LLM to summarize or contextualize each patient's trajectory (Record2Vec [3], TimeCP [4]). These methods and STEP-CTS both delegate clinical interpretation to an LLM, but our concepts are checked once offline. A summary also takes 12.8–19.6 s per patient against 171 ms for ours (Appendix H).

Weakness 1 and Question 2 (selector objective). You are right about Eq. (4), and thank you for pointing this out. There, zijsz^s_{ij} says whether eije_{ij} is in SisS_i^s after the top-K truncation, and that does not give the gradient of the sampling policy. The equation was written incorrectly. Our method uses the Bernoulli samples drawn before the truncation:

  1. Sample aij∼Bernoulli(qij)a_{ij}\sim\mathrm{Bernoulli}(q_{ij}) independently for each candidate.
  2. Among the units with aij=1a_{ij}=1, keep the top K=30K=30 by score rijr_{ij}. If fewer than 5 were sampled, add the highest-scoring units up to 5 (this is also now stated in Section 3). This is SisS_i^s.

The selector loss uses aija_{ij}, not whether the unit ended up in SisS_i^s:

LselSC=−1∣B∣∑i∈BA^i1Ni∑j=1Ni[aijlog⁡qij+(1−aij)log⁡(1−qij)].\mathcal L_{\mathrm{sel}}^{\mathrm{SC}} = -\frac{1}{|B|}\sum_{i\in B}\hat A_i \frac{1}{N_i}\sum_{j=1}^{N_i}\big[a_{ij}\log q_{ij} + (1-a_{ij})\log(1-q_{ij})\big].

Here A^i\hat A_i is the advantage of Eq. (3). Once aia_i is sampled, SisS_i^s is fixed by aia_i and the current score order, so this is the score-function estimator for sampling followed by truncation, with the greedy set as baseline (Appendix G.2).

Your comment also made us curious how much this term matters for the results, so during the rebuttal we additionally trained two more versions of the selector loss (Table R5):

  • (A) Eq. (4) as submitted. In the log-probability, the units kept in SisS_i^s get log⁡qij\log q_{ij} and all other units get log⁡(1−qij)\log(1-q_{ij}), including those that were sampled and then cut by the truncation. This is the loss you described.
  • (B) Probability of the kept set. Same as ours except when KK or more units are sampled: then every unit that scores below the KK-th kept unit gets no gradient.

Our loss gives the highest AUPRC on every dataset. (A) is lower by 3.2 / 6.2 / 1.8 points on P2012 / MIMIC-III / P2019, so the loss as it was printed would have hurt. (B) is very close to ours. We do not claim that ours is better than (B): the two estimators perform similarly, and only (A) seems clearly worse. On AUROC, (A) is 1.9 points lower on MIMIC-III; the other differences are below 0.7 points, and on P2019 both variants are slightly above ours.

Table R5. Three versions of the selector loss; everything else is unchanged. Mean ± s.d. over 3 seeds.

Selector lossP2012 AUROCP2012 AUPRCMIMIC-III AUROCMIMIC-III AUPRCP2019 AUROCP2019 AUPRC
Ours (sampled units, corrected Eq. (4))0.8561±0.00170.5274±0.00490.8563±0.00240.5025±0.00560.9115±0.00160.5004±0.0157
(A) Eq. (4) as submitted0.8496±0.00330.4951±0.01590.8377±0.01880.4408±0.02140.9147±0.00070.4827±0.0124
(B) Probability of the kept set0.8502±0.00100.5132±0.01330.8543±0.00010.4979±0.00500.9140±0.00240.4951±0.0071

In the revision we replaced zijsz^s_{ij} with aija_{ij} in Eq. (4) and added this comparison as Table G2 in the new Appendix G.2.

AuthorsOpenReview ↗

Response to Reviewer ujxE (Part 2/2)

Question 3 (LLM-injected semantics versus evidence selection). Both the evidence selection and the LLM-written concepts contribute to the gain. The ablations separate them (mean AUPRC on P2012 / MIMIC-III / P2019).

Selection. With the concepts held fixed, replacing our selector with rule-score, recency, or random ranking lowers AUPRC on every dataset; even the best alternative ranking is 3.7 / 3.3 / 1.3 lower (Table 2, panel B). The selector never reads the concept text; concepts reach the selector only as a binary availability flag (Section 3). With no LLM-written concept text at all, STEP-CTS is already above the best text baseline on P2012 and P2019 (+2.3 / +9.5) and equal to it on MIMIC-III (Tables 1 and 2). Record2Vec and TimeCP, which generate a per-patient summary with the same Qwen3.5-27B, score lower than STEP-CTS without concepts on all three datasets (Table 1).

Concepts. Concepts add a further 2.8 / 3.8 / 2.2 (Table 2, panel A). Labeling more rules does not help: removing abstention raises concept coverage from 26-43% to 86-90% of rules and lowers AUPRC by 2.9 / 1.7 / 3.4 (Appendix Table D1). Qwen3.5-27B writes the concepts once per rule, without patient records, the task, or outcome labels. Claude Sonnet 5 is not part of the model; it only audits the concepts afterwards (Appendix E.2).

During the rebuttal we additionally used these audit grades for one further check (Table R1 in our response to Reviewer qYme). Training with only the 5,215 concepts graded as entailed gives AUPRC of 0.5311 / 0.5028 / 0.5001, against 0.5274 / 0.5025 / 0.5004 for the full model. Showing only the unsupported concepts at test time (0.5086 / 0.4814 / 0.4735) is no better than showing none (0.5090 / 0.4949 / 0.4763).

[1] Fadlon, D., Dov, D., Bennett, A., Heller-Miron, D., Levy, G., Bar, K., and Weiss-Meilik, A. Decode Like a Clinician: Enhancing LLM Fine-Tuning with Temporal Structured Data Representation. IJCNLP-AACL, 2025.

[2] Jain, S., Wiegreffe, S., Pinter, Y., and Wallace, B. C. Learning to Faithfully Rationalize by Construction. ACL, 2020.

[3] Ji, Z., Sun, Y., Amaral, A., Goldenberg, A., and Krishnan, R. G. Can We Generate Portable Representations for Clinical Time Series Data Using LLMs? ICLR, 2026.

[4] Lee, G., Yu, W., Shin, K., Cheng, W., and Chen, H. TimeCAP: Learning to Contextualize, Augment, and Predict Time Series Events with Large Language Model Agents. AAAI, 2025.

[5] Lei, T., Barzilay, R., and Jaakkola, T. Rationalizing Neural Predictions. EMNLP, 2016.

Reviewer_8N5eOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors create a pipeline whereby they discretize EHR data into temporal windows and generate features within each window. They then use an LLM to learn how to describe each feature and identify the most important “rules” for a given outcome/patient that they present to a clinician. The primary contribution is the creation of a selectable and deterministic training process.

Critical strengths
  1. The authors include a significant number of ablation studies that support the necessity of each step in their architecture. They also include a comprehensive set of comparator models form prior work.
  2. The deterministic nature of the evidence generation is important for clinical application.
Critical weaknesses
  1. Lack of clarity and use of inconsistent terminology: from the methodology, it is not clear what “threshold rules” are, how they differ from features or tokens (both of which are more standard ML terminology), and what an “inventory” of trees are. The authors also use both “rules” and “predicates” without specifying the difference. The graphic of the paper is also busy and confusing.
  2. The authors do not treat the XGBoost model “fairly”: they use only 30 trees, which is consistent with the 30 evidence budget. However, there are aspects of training (e.g. the temporal evidence verbalizer) that potentially take up more training budget. It might be helpful to see what model performance looks like under a more complex baseline model. Additionally, the use of SHAP values does not seem to be a one-to-one comparison with source-traceable predicates: SHAP values are specifically designed to communicate importance as opposed to use in the model. Another possible way to handle this would be two-part training of the XGBoost baseline, where you train on all features, isolate the most important features, and retrain only on those; this way, you could provide a list of the features/rules used to train the baseline model as opposed to the SHAP values.
  3. In the clinician validation, the authors emphasize that part of the workflow is “selected/important” evidence. Given that this is work about source-traceable predicates (as opposed to being about feature importance), the clincician evaluation loses significant validity. More broadly, the authors do not define (or justify) any desiderata regarding what a trace should look like or contain. The instructions to the clinicians were hyper-specified to the evidence format generated by this work, and it seems like it would be relatively straightforward to map other evidence formats to this format (for example, using deterministic tools like MetaMap to “cluster” concepts, as this was something that the clinicians liked about your approach).
Questions for authors
  1. [Clarification] Is it the case that the self-critical selector learning is a way to decide which features are used to run inference for a certain patient? Relatedly, at inference time, does the model see any of the features outside of the selected set of top-K features? Note that situating your work within the context of feature selection could be helpful.
  2. [Clarification] In the methods, v_j is initially introduced as a predictor variable. Later, v_i,j is stated to be a cached embedding -- please clarify if these are the same variables or not.
  3. [Clarification] What are the specific desiderata of what makes “clinician-readable” threshold evidence? How did you arrive at these principles?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 8N5e (Part 1/2)

We thank you for the feedback. We fixed the problems you pointed out, including a redrawn Figure 1, and where points were already covered in the submission but hard to find, we revised the presentation so that they are easier to find. The paper is clearer as a result, and we are grateful for that. All changes in the revised PDF are marked in blue. We first restate the method briefly and then summarize our responses:

  • The method in brief. Fixed rules turn each patient’s time series into evidence units. Each unit consists of text predicates, and each predicate states a threshold on one statistic of one variable in one time window. An LLM, run once per rule and never on patient data, attaches a clinical concept where the predicates support one; the LLM does not select evidence or make the prediction. A selector trained jointly with the LM reader keeps 30 units per patient, and the reader predicts only from those 30 units and the static covariates, so what a clinician inspects is what the model read. Unlike serialization [1, 2], this evidence carries a clinical interpretation; unlike generated summaries [3, 4], each concept is tied to its supporting predicates and source time window, so each interpretation can be checked once for all patients, and no text is generated per patient at training or test time.
  • Clarity problem (Weakness 1, Question 2). Fixed in the revision: four terms are defined once in Section 3 and used consistently throughout the main paper, one symbol is renamed, and Figure 1 is redrawn.
  • XGBoost (Weakness 2). Besides the 30-tree XGBoost, Table 3 already includes a tuned XGBoost (“XGB-WindowStats, tuned”) and models built specifically for irregular EHR time series. We report where they are better than ours.
  • Clinician study (Weakness 3, Question 3). All four explanations were rated blind on the same three scales, and the results do not uniformly favor our method.

Weakness 1 and Question 2 (terminology and notation). We agree; this was our fault. The revision defines four terms once in Section 3 and uses only these:

  • Feature: one window statistic of one variable (e.g., the last temperature in [5-7 h]).
  • Predicate: a threshold on one feature, written as text ("last temperature at least 38 °C").
  • Rule: a conjunction of predicates, that is, one root-to-leaf path of a tree. The rules are learned once on the training split and then fixed (the rule set).
  • Evidence unit: a rule that holds for a patient in a given time window, with its attached concepts.

So a rule is not a feature but a condition on several features, and tokens are only the subwords of its text. These terms replace "path", "leaf", "clause", "predicate set", "threshold rule", and "inventory" throughout the main paper. Figure 1 is redrawn around the four terms: it follows one patient from a feature to its predicate, rule, and evidence unit, through TEV, ES, and the LM reader, without the undefined symbols of the old figure. On Question 2, the two symbols are different: vjv_j is the measured variable of observation jj, and vijv_{ij} was the embedding of a unit's predicate text. We renamed the latter dijd_{ij}.

Weakness 2 (XGBoost baseline and SHAP).

  • A more complex baseline. Table 3 has two XGBoost rows. The second, "XGB-WindowStats, tuned" (XGB-tuned), selects the number of trees on validation (200 / 100 / 100 on P2012 / MIMIC-III / P2019), and Table C2 gives the full sweep up to 200 trees. Section 5 states that it "surpasses STEP-CTS on P2012 and on P2019 AUROC". Table 3 also compares with models built specifically for irregular EHR time series (e.g., GRU-D, STraTS, DuETT).
  • Training budget. The two XGBoost rows serve two purposes. The 30-rule row (XGB-30) is matched on what the final predictor reads per patient: 30 rules against our 30 units. As Appendix C states, it is not matched to ours on computation. XGB-tuned is not constrained to the 30-rule budget; its tree cap is selected by validation AUROC from a sweep up to 200 trees.
  • SHAP. SHAP is not used for training or prediction. SHAP-to-Text appears only in the clinician study, as the explanation method of the already trained XGB-30, modeled on clinician-facing SHAP displays [5]. We agree that SHAP communicates importance, not use, and that contrast is why it is included: Section 5 notes that it "shows which features a separate predictor weighted, not what the predictor read."
  • The suggested alternative. We did not run this variant. Its goal is a list of the rules that the baseline actually uses, and XGB-30 already gives one without retraining: each patient follows one rule in each of its 30 trees. We gave these 30 rules, with concepts attached in the same way, to the same LM reader in place of our selected units. This is 3.2-4.8 AUPRC points below STEP-CTS (Table R3 below; Table C3 in Appendix C.1 of the revision).
AuthorsOpenReview ↗

Response to Reviewer 8N5e (Part 2/2)

Table R3 (excerpt; the full table is in our response to Reviewer g1uV). Mean ± s.d. over 3 seeds.

SettingP2012 AUROCP2012 AUPRCMIMIC-III AUROCMIMIC-III AUPRCP2019 AUROCP2019 AUPRC
STEP-CTS (Ours)0.8561±0.00170.5274±0.00490.8563±0.00240.5025±0.00560.9115±0.00160.5004±0.0157
XGB-30 (Table 3)0.8482±0.00220.5002±0.00460.8391±0.00050.4290±0.01060.9051±0.00110.4208±0.0006
Rules of XGB-30 → LM reader, predicates with concepts0.8371±0.00430.4796±0.01650.8435±0.00090.4623±0.00060.9136±0.00370.4689±0.0045

Weakness 3 and Question 3 (clinician study; what a trace should contain).

  • "Selected/important evidence." In the clinician instructions (Appendix M.3), this phrase names what each of the four explanations displays, not ours in particular. Traceability is rated directly: "Can you identify the original patient data on which the presented evidence is based?"
  • What a trace should contain. We arrived at the requirements from where the two existing interfaces fall short: serialization shows source values without interpretation, and generated summaries interpret but can omit or distort source measurements (Section 1). They are the property columns of Table 1: source traceability ("a deterministic pathway to source variables and time windows") and clinical interpretation ("findings beyond numeric values, trends, or thresholds"). Concretely, evidence is produced deterministically and names its variable, statistic, and time window. A clinical concept is shown only with the predicates that support it. Section 3 of the revision now states these requirements and their motivation explicitly.
  • Are the scales tailored to our format? Each scale measures a property that an existing interface already offers: traceability for serialization, interpretation for generated summaries (Table 1). The ratings reflect this: one clinician rated Decode [1] above ours on traceability, and the generated summary above ours on interpretation and usefulness (Table F1(a)). All four explanations were rated blind on the same scales.
  • MetaMap. MetaMap maps words in a text to UMLS concepts; it does not interpret numbers. Applied to Decode or SHAP-to-Text, which list variables and values, it would group by variable ("heart rate"), not by clinical state ("tachycardia"). Naming the state needs a cutoff on the value, a rule step beyond MetaMap, and that step is what our predicates and concepts provide, only for the units that the model selected and read. Applied to a generated summary, it could group the stated findings but would not restore their link to the source measurements.

Question 1 (selector and feature selection). Yes, the selector decides which evidence units are used for each patient: it scores every candidate unit and keeps K=30K=30. And no, at inference the LM reader sees nothing from the time series outside these 30 units: it predicts from the static prefix and the selected units only. Contribution 2 states that "the selected units are the reader's entire time-series input", and Figure 2 tests it by deleting units. As you suggest, the revision adds the connection to feature selection in Section 2.

We hope the revised terminology and Figure 1 make the paper easier to follow, and that the points above resolve the concerns about the baselines and the clinician study. If so, we would be grateful if you would consider updating your score.

[1] Fadlon, D., Dov, D., Bennett, A., Heller-Miron, D., Levy, G., Bar, K., and Weiss-Meilik, A. Decode Like a Clinician: Enhancing LLM Fine-Tuning with Temporal Structured Data Representation. IJCNLP-AACL, 2025.

[2] Gao, Y., Myers, S., Chen, S., Dligach, D., Miller, T. A., Bitterman, D., Churpek, M., and Afshar, M. When Raw Data Prevails: Are Large Language Model Embeddings Effective in Numerical Data Representation for Medical Machine Learning Applications? Findings of EMNLP, 2024.

[3] Ji, Z., Sun, Y., Amaral, A., Goldenberg, A., and Krishnan, R. G. Can We Generate Portable Representations for Clinical Time Series Data Using LLMs? ICLR, 2026.

[4] Lee, G., Yu, W., Shin, K., Cheng, W., and Chen, H. TimeCAP: Learning to Contextualize, Augment, and Predict Time Series Events with Large Language Model Agents. AAAI, 2025.

[5] Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., and Lee, S.-I. Explainable Machine-Learning Predictions for the Prevention of Hypoxaemia during Surgery. Nature Biomedical Engineering, 2018.

AuthorsOpenReview ↗

Follow-up to Reviewer 8N5e

Dear Reviewer 8N5e,

Thank you again for your review. We would like to check whether our response and the revised PDF address your concerns. In short:

  • Weakness 1 (clarity). The revision defines four terms (feature, predicate, rule, evidence unit) in Section 3 and uses only these, and Figure 1 is redrawn and simplified.
  • Weakness 2 (XGBoost). Besides the 30-tree XGBoost, Table 3 of the submission already includes a tuned XGBoost with up to 200 trees, and we report where it is better than ours. The revision also gives the rules of the 30-tree XGBoost to our LM reader, and the rules of the tuned XGBoost to our Evidence Selector and LM reader (Table C3).
  • Weakness 3 (clinician study). All four explanations were rated blind on the same three scales. Section 3 now states the two requirements for our evidence: source traceability and clinical interpretation.

If any point remains unclear, we would be glad to answer it within the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the Area Chair

We thank all reviewers for their constructive comments, and Reviewer g1uV and Reviewer 8N5e for raising their scores after our response. All text changes in the revised PDF are marked in blue.

[1] Paper Summary and Contributions

STEP-CTS has three parts: a Temporal Evidence Verbalizer (TEV) turns each patient's irregular clinical time series into text evidence, an Evidence Selector (ES) learns to select a set of this evidence, and a language model (LM) predicts from the selected evidence, which is its entire time-series input. Numerical models for irregular time series predict well but show no readable text evidence. Text-based methods show evidence that lacks either clinical interpretation (serialization) or source traceability (LLM-generated summaries). What is new in STEP-CTS is that the evidence the model reads has both. It is also more predictive than every evaluated text-based baseline and competitive with the numerical models.

[2] Strengths Highlighted by Reviewers

  1. A significant problem with practical value (ujxE).
  2. Evidence that is what the model reads. The source-linked bottleneck is "the strongest part of the idea" and "connects the displayed evidence to what the model actually consumes" (qYme), and the deterministic evidence generation is important for clinical application (8N5e).
  3. A transparent, well-controlled evaluation (g1uV) with a sound experimental design (ujxE), substantial controls (qYme), and comprehensive comparators and ablations (8N5e): "I am confident in the numbers that are reported" (g1uV).

[3] Main Concerns and How They Were Addressed

1. Is the gain driven solely by the outcome-supervised tree rules, not the text evidence? (g1uV, score 2→3) No. With the tree rules held fixed, replacing the learned selector with the best alternative ranking, or the language model with an MLP over rule IDs, lowers AUPRC on every dataset (Table 2). The revision adds two controls (Table C3). The best text-based baseline, given our outcome-supervised rules and selection, improves but stays below STEP-CTS in AUPRC on every dataset. The rules of the tuned XGBoost, given to our selector and language model in place of our own rules, perform worse than STEP-CTS on every metric, although the tuned XGBoost (up to 200 trees, already in the submission; 8N5e) alone is a far stronger predictor than the tree models behind our rules (Tables 3 and C1). So supervision explains part of the gain, and the text evidence explains the rest.

2. Are the LLM-written concepts faithful, and do they drive the gain? (qYme, g1uV, ujxE) Like the LLM-generated summary baselines, STEP-CTS delegates clinical interpretation to an LLM, so faithfulness is not guaranteed. But the concepts are written once per rule, offline, and the LLM writes nothing at test time. They can be checked once, in advance, for all patients, whereas a per-patient summary needs a separate check for every patient. We ran this check (53.9% entailed, 41.5% weak, 4.5% unsupported), and the predictive gain comes mainly from the faithful concepts: training with only the concepts graded as entailed preserves the AUPRC of the full model on every dataset, while the unsupported concepts give no gain on their own (Table E3). The concepts are not the only source of the gain: without them, STEP-CTS is on par with or above the best text-based baseline (Tables 1 and 2).

3. Does the clinician study support the claims about clinical interpretation? (qYme, g1uV, 8N5e) To isolate the benefit of the concepts (qYme), we added a paired comparison of the same selected evidence with and without concepts: both clinicians and both LLM judges rate the version with concepts higher on clinical interpretation and usefulness (Tables F2, F3). The revision now claims only "comparable to generated summaries" on interpretation and describes the study protocol more clearly (g1uV), and Section 3 now states what a trace should contain (8N5e).

4. The selector loss in Eq. (4). (ujxE) The reviewer is right, and we corrected the equation by replacing one symbol (zijs→aijz^s_{ij} \to a_{ij}). Only the written equation was wrong: our implementation, and so every reported result, already used the correct loss. We also ran two more experiments (Table G2). One, which gives zero gradient to the units cut by the top-K truncation, performs similarly to ours; the other, the equation as originally printed, performs worse.

5. Terminology and Figure 1 are hard to follow. (8N5e, score 1→2) The revision defines four terms (feature, predicate, rule, evidence unit) in Section 3 and uses only these throughout the main paper. Figure 1 is redrawn with the same four terms.

6. Stability and reporting details. (g1uV, score 2→3) Relearning the rules with three new seeds gives similar results on P2012 (Table B7). We corrected the input length in Table K1 and the cohort exclusion counts in Table A1(c).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.