acceptodds
Under review as a conference paper at ICLR 2027

TRIAGE: Dialectical LLM Reasoning for Explainable Risk Prediction on Irregularly Sampled Medical Time Series

Abstract

Clinical early warning systems built on irregularly sampled medical time series (ISMTS) from electronic health records must deliver continuous risk scores for patient triage as well as interpretable rationales that clinicians can verify. Large language models (LLMs) are uniquely positioned for both, deriving risk from their output probabilities and rationales from their medical knowledge. However, we find that conventional LLM reasoning collapses graded risk into overconfident predictions and thereby undermines the cross-patient comparability on which triage depends. We refer to this failure mode as and identify two underlying behaviors: early commitment to a single outcome, and one-sided reasoning that focuses only on the evidence for that outcome. To address this, we propose , a framework that trains an LLM to reason dialectically over competing clinical outcomes by eliciting outcome-specific rationales. This dialectical formulation mitigates risk polarization, enabling a single LLM to jointly provide explicit clinical rationales and risk scores comparable across patients. Across five ISMTS benchmarks, TRIAGE improves mean AUPRC by 17.0% and reduces mean calibration error by 82.8% relative to the competitive LLM-based baseline, while surpassing the strongest ISMTS baseline by 3.5% in mean AUPRC.

Primary area:
learning on time series and dynamical systems
Keywords:
clinical time series, irregular time series classification, clinical risk prediction, large language models, explainable AI, medical AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pvxzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TRIAGE for clinical risk prediction from irregularly sampled medical time series. It uses outcome-specific rationales to address risk polarization, followed by SFT and GRPO with a batch-level ranking reward. Experiments on five benchmarks report improvements in prediction and calibration.

Critical strengths

The paper addresses an important clinical prediction problem. The observation of risk polarization is interesting, and the experiments cover multiple clinical benchmarks.

Critical weaknesses
  1. I am not convinced that the method offers sufficient novelty. The two outcome-specific rationales are generated independently and concatenated, rather than explicitly compared or revised. The subsequent SFT and GRPO largely follow existing methods. It is also unclear what specifically addresses irregular sampling.

  2. I find the baseline presentation potentially misleading. Table 3 highlights only explainable methods, although specialized models perform better on some predictive metrics (e.g., P19 AUPRC: 56.6 vs. 53.8). The caption explains this choice, but I do not think explanation availability justifies different rankings for the same metrics. Table 4 also excludes specialized models, while Table 13 shows higher overall explanation ratings for GPT-5.1 than TRIAGE (4.68 vs. 4.48).

  3. The ablations are not sufficiently convincing. Answer-only SFT already achieves 53.4 AUPRC versus 56.4 for TRIAGE SFT, and the one-sided baseline uses a different scoring procedure. Appendix G.1 also shows that sample-level rewards already reduce ECE from 0.246 to 0.087, compared with 0.079 for batch-level rewards. The individual contributions remain unclear.

  4. I also noticed differences in the P12 results. Tables 3–4 report AUPRC 59.0, ECE 0.044, and Brier score 0.089, while Appendix G.1 reports 64.35, 0.079, and 0.037. The SFT reference results also differ. These differences need a clearer explanation.

Questions for authors
  1. What fundamentally distinguishes TRIAGE from outcome-conditioned reasoning supervision followed by GRPO? What is specific to irregular time series?

  2. Why are specialized models excluded from the highlighted rankings in Table 3 and the calibration comparison in Table 4?

  3. Could the authors provide more controlled ablations to separate the effects of dialectical reasoning and the batch-level reward?

  4. Could the authors clarify the different P12 results, particularly the checkpoint selection and evaluation settings?

11 comments
AuthorsOpenReview ↗

Response to Reviewer pvxz (1/3)

We thank the reviewer for the careful review and detailed feedback. We appreciate the recognition that our work addresses an important clinical prediction problem, that our observation of risk polarization is interesting, and that our experiments cover multiple clinical benchmarks.


[W1-1, Q1-1] Novelty of the TRIAGE Framework

We thank the reviewer for raising this important question about the novelty of TRIAGE relative to existing SFT and GRPO approaches. Our contribution centers on a reasoning formulation in which a single LLM produces a clinical risk score that its own rationale explains. SFT and GRPO are standard training methods that we use to instill this formulation. We would like to clarify the line of work that TRIAGE belongs to and what it contributes to that line.

1. Line of work: LLM reasoning for medical time series. Specialized architectures for irregularly sampled medical time series (ISMTS) mostly come from the irregular time series literature, where they are used as baselines. They focus on modeling irregularity and produce strong risk scores, but they give no rationale. Medical decisions are high-stakes, and models that support them are expected to be explainable as well as accurate [1, 2]. LLM-based methods address the same clinical prediction tasks while giving a natural-language rationale for each prediction. TRIAGE belongs to this LLM-based line, so we do not claim a new technique for handling irregularity. Because most other methods in this line output hard labels, they have rarely been compared directly with the specialized architectures, even though both use the same datasets. TRIAGE produces a risk score together with its rationale, so we fill this gap by reporting the specialized architectures alongside our main comparison with LLM-based methods (Table 3).

2. Contribution: a rationale and a risk score from one model. Early warning and triage need a cross-patient comparable continuous risk score, but existing LLM-based methods output hard labels or hand scoring to a separate predictor (Table 1). We designed the reasoning-and-answer structure for the clinical decision process, so that one model provides both the rationale and the score. Our preliminary study (Section 4) reveals the risk polarization problem that collapses the predicted probability due to early commitment and one-sided reasoning. Since a single ICU trajectory often carries signals of both deterioration and stabilization (Arnold et al., 2009; Hernandez et al., 2014), TRIAGE has one LLM reason about both outcomes and then produce the cross-patient comparable risk score, all in a single decoding pass. To the best of our knowledge, prior LLM work on ICU data has not used such a design.

3. Training: a probability-level reward that compares patients within a batch. The training procedure also differs from SFT followed by standard LLM GRPO in that each rollout’s reward depends on the predictions for other patients. The model outputs a continuous probability, and we define the reward directly on this probability. From one patient alone, however, it is difficult to judge whether a probability is appropriate for cross-patient discrimination, because the same value can be too high or too low depending on the probabilities assigned to other patients. We therefore designed a batch-level reward based on the margin between a rollout’s probability and the probabilities assigned to patients with the opposite outcome in the same batch (Sec. 5.2).

[1] Amann, Julia, et al. “Explainability for artificial intelligence in healthcare: a multidisciplinary perspective.” BMC Medical Informatics and Decision Making 20.1. 2020.

[2] Tonekaboni, Sana, et al. “What clinicians want: Contextualizing explainable machine learning for clinical end use.” Machine Learning for Healthcare Conference. 2019.

AuthorsOpenReview ↗

Response to Reviewer pvxz (2/3)

[W1-2, Q1-2] Handling of Irregular Sampling

As clarified in [W1-1, Q1-1], the goal of TRIAGE is to have an LLM produce well-calibrated risk scores with explanations when it is given irregular medical time series. Its text input, however, can represent irregular sampling directly, because a textualized record only needs to contain the values that were actually observed. Following the set-based encoding of Horn et al. (2020), we serialize each observed measurement together with its timestamp (Sec. 5.1), so the input needs no imputation or resampling and the sampling pattern is preserved. Given this input, the LLM can interpret the sampling pattern on its own, and some of its rationales do so in ways that resemble dedicated components of ISMTS architectures (pvxz-Table 1). This behavior appears without any irregularity-specific design, so we see LLM reasoning as a promising direction for irregular clinical time series.

pvxz-Table 1. Irregularity signals modeled by ISMTS architectures, and example TRIAGE rationales that draw on the same signals (P12).

Irregularity signalTRIAGE rationale (RecordID)
Time since last observation (GRU-D)“Mechanical ventilation is used early and then not documented beyond 31.2 h, … implying successful weaning” (143187)
Observation frequency / intensity (IP-Net)“Electrolytes (Na, K, Mg, HCO3) … are repeatedly measured, indicating close monitoring” (153430)

[W2 & Q2] Presentation of specialized ISMTS models in Tables 3 and 4

We thank the reviewer for carefully examining our baseline comparisons and highlighting concerns about the presentation of specialized ISMTS models. We would like to re-emphasize that effective clinical decision support must provide both high predictive performance and explainability. Specialized ISMTS models predict well, but they do not explain their predictions, which makes them hard to adopt in clinical practice [1, 2]. Post-hoc explanations do not resolve this, as the clinicians in our study found them of limited use (Appendix F.1, Table 13). Our primary target for direct comparison is therefore explainable LLM-based systems.

  • Highlighted rankings in Table 3. We listed the specialized models as high-performance references. We agree that the highlighting could be misread, and we revised Table 3 to highlight the top performance across all baselines.
  • Calibration comparison in Table 4. Table 4 has a different purpose from Table 3. It evaluates whether a probability that an LLM produces together with its reasoning is well calibrated, and this comparison is meaningful only among LLM-based methods. A specialized model outputs a probability without reasoning, and that probability can be freely recalibrated or otherwise adjusted post hoc, so it falls outside this comparison (the full results, requested by Reviewer kw4Q, are in kw4Q-Table 4). For reference, pvxz-Table 2 reports the specialized models averaged over the five benchmarks, and TRIAGE still has the lowest ECE and Brier score.
  • Explanation ratings in Table 13. In the original Table 13, GPT-5.1 received a higher explanation rating than TRIAGE (4.68 vs. 4.48). Its explanations are informative, but it does not produce a risk score that discriminates between patients (Table 3). TRIAGE reaches a comparable explanation rating together with higher predictive performance. It does so with a 4B-parameter model, far smaller than a proprietary frontier model such as GPT-5.1.

pvxz-Table 2. Predictive performance and calibration of the specialized ISMTS models and TRIAGE, averaged over the five benchmarks.

Modelavg. AUROCavg. AUPRCavg. ECEavg. Brier
GRU-D86.755.50.1900.136
KEDGN86.855.40.1910.136
STraTS87.055.00.1830.136
TRIAGE$_{\text{SFT}}$86.855.30.1800.138
TRIAGE87.357.40.0380.077
AuthorsOpenReview ↗

Response to Reviewer pvxz (3/3)

[W3-1] Reasoning-structure ablation: gap over answer-only SFT and scoring of the one-sided baseline

We thank the reviewer for the careful examination of our reasoning-structure ablations, particularly the comparison with answer-only SFT and the scoring procedure of the one-sided baseline. We clarify these points below to better explain the contribution of our reasoning design.

Answer-only SFT is not practical for real-world workflows because it lacks explainability. In contrast, TRIAGE$_{\text{SFT}}$ provides explainability and also outperforms it by 3.0 AUPRC points. Furthermore, the one-sided rationale baseline inevitably requires a different scoring procedure. One-sided rationales suffer from risk polarization, which drives the predicted probabilities to extreme values (Sec. 4). As a result, under the same scoring as TRIAGE, its AUROC and AUPRC fall to 75.1 and 33.1 (pvxz-Table 3). To avoid understating this baseline’s performance, we instead scored it by the fraction of 10 sampled responses that predict the positive label. This costs 10 decoding passes per patient and favors the baseline, which still performs substantially worse than both the answer-only model and TRIAGE$_{\text{SFT}}$.

pvxz-Table 3. Ablation of the reasoning structure on P12 (Table 5).

Method (P12)AUROCAUPRC
Answer-only86.4±1.553.4±2.2
One-sided rationale (10 samples, averaged)83.8±1.043.1±3.6
One-sided rationale (same scoring as TRIAGE)75.1±1.033.1±3.0
TRIAGE$_{\text{SFT}}$86.9±1.056.4±1.9

[W3-2] Reward ablation: calibration of sample-level vs. batch-level rewards (Appendix G.1)

Thank you for this observation. Compared to the sample-level reward, batch-level reward reduces ECE and Brier score by 21.3% and 9.2%, respectively. While we consider these reductions meaningful, we agree that the batch-level reward mainly improves discrimination. We revised the manuscript (Sec. 5.2 and Appendix G.1) to clarify that, although the batch-level reward is designed to improve discrimination performance, the observed improvements in calibration are secondary benefits rather than direct objectives of its design.


[Q3] Controlled ablations that separate dialectical reasoning from the batch-level reward

Thank you for the constructive suggestion. We agree that additional ablations would help clarify the individual contributions of dialectical reasoning and batch-level rewards. To separate the two effects, we ran one additional experiment: a one-sided reasoning model trained with the same batch-level reward and RL setup as TRIAGE, on the same P12 split. It differs from TRIAGE only in the reasoning format.

pvxz-Table 4. Ablation on P12 (single split). Each variant removes one component from TRIAGE.

MethodAUROCAUPRC
w/o Dialectical Reasoning87.2360.64
w/o Batch-level Reward88.3762.48
TRIAGE (Ours)89.2364.35

Removing dialectical reasoning lowers AUROC by 2.00 and AUPRC by 3.71. Both dialectical reasoning and the batch-level reward contribute to the performance of TRIAGE.


[W4 & Q4] Different P12 results between Tables 3-4 and Appendix G.1

We thank the reviewer for the careful reading and for raising this inconsistency. The gap comes from a difference in the evaluation setting and a typo in Appendix G.1. We explain both below, together with our checkpoint selection protocol.

  • Evaluation setting: Tables 3-4 report the average over all 5 P12 splits, whereas the ablation in Appendix G.1 was run on a single split because of its computational cost. This also applies to the SFT reference, which is why its numbers differ as well. We have added an explanation of this in Appendix G.1 to clarify this.
  • Typo in ECE and Brier score: We thank the reviewer for figuring this out. The ECE and Brier score values in Appendix G.1 were swapped. The correct values are ECE 0.037 and Brier score 0.079, close to the 5-split averages in Tables 3-4 (0.044 and 0.089). We have corrected them in the revised version.
  • Checkpoint selection: We run validation every 0.25 epochs for SFT and every 15 rollout steps for RL, and report test results at the checkpoint with the best validation AUPRC. Consistent with Kang et al. (2026), which shows that a higher SFT score does not reliably lead to better performance after RL, we empirically initialize RL from the best SFT checkpoint within the first 2 epochs instead of the overall best SFT checkpoint. See Appendix C.4 for details.
Reviewer_25nTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The goal of the paper is to develop a model that advances explainable risk scoring of triage in ICU settings. It identifies the risk polarization problem, which can be rooted in the auto-regressive nature of LLMs that favors the most convincing story over a strictly fact-based balancing by early commitment to one outcome and one-sided reasoning. The authors propose a simple and effective solution: prompting the LLM to reason independently and without a final verdict on both possible outcomes, death or survival, and deriving the risk score from combining the resulting output logits of these two dialectic scenarios. They use supervised fine-tuning on outcome-conditioned rationales distilled from a strong teaching model and reinforcement learning that includes batch-level rewards for ranking patients against each other. The proposed model performs well against benchmarks used in scientific literature. Three contributions stand out: the identification and quantification of the risk polarization problem itself, the implementation of the dialectic design that refuses any verdict before the final token and, to some extent, also the batch-level ranking. The validation measures seem sound. The drawbacks and limitations are well expressed. However, what I miss is the comparison to a baseline that reflects the state of the art in clinics, which often work with simple, robust, and at the same time explainable rules-of-thumb that, in many cases, perform astonishingly well.

Critical strengths

To me, this is really the way the two-sided reasoning is implemented. In the paper, the model is not allowed to write a concluding verdict for any of the two possible outcomes. This seems a bit counterintuitive at first but makes sense in the context and it is supported by the analysis. This omittance of a conclusive summary might be a transferable nugget.

Critical weaknesses

The work does not benchmark itself against the practical state of the art in clinics, and clinical validation is limited only to a 30-case two-clinical rating. So, its main contributions are bound to be on the methodological side, which consists of a well-executed recombination of known approaches with one generally fresh insight mentioned in strengths.

Questions for authors
5 comments
AuthorsOpenReview ↗

Response to Reviewer 25nT

We sincerely thank the reviewer for the thoughtful review and positive evaluation of our work. We particularly appreciate the recognition that omitting concluding verdicts from the two-sided reasoning is supported by our analysis and offers a potentially transferable insight.


[W1-1] Comparison with the practical state of the art in clinics

We thank the reviewer for the constructive feedback on strengthening the clinical relevance of our work by grounding our evaluation in established clinical practice. Severity scores have filled that role for a long time because they are simple and interpretable. We ran this comparison on P12 and report it in 25nT-Table 1. SAPS I and SOFA come with the dataset. We also add SAPS II and APACHE II, two other widely used scores that we can compute from the recorded variables. All four scores show modest discrimination, with AUROC between 64.2 and 72.6 and AUPRC between 25.5 and 36.2. When TRIAGE is set to the same specificity as each score, its sensitivity is 26 to 41 percentage points higher.

25nT-Table 1. Clinical severity scores versus TRIAGE on P12. All values are percentages averaged over five splits. We compute SAPS II and APACHE II from the recorded variables. In the last two columns, the first number is the score's sensitivity at a specificity of at least 90% (or 80%), and the number after the arrow is TRIAGE's sensitivity at that specificity.

ScoreAUROCAUPRCSens. @ Spec. ≥ 90% (score → TRIAGE)Sens. @ Spec. ≥ 80% (score → TRIAGE)
SAPS I65.825.521.7 → 55.833.4 → 73.5
SOFA64.226.222.0 → 57.031.4 → 71.0
SAPS II72.636.235.2 → 61.348.7 → 77.3
APACHE II66.928.222.9 → 57.935.2 → 76.4

[W1-2] Scale of the clinical validation

We agree that a validation with 30 cases and two clinicians is limited, and that a fuller study needs more cases and more clinicians. Expert assessment is costly, so for this revision we extended the study with the same two clinicians, who rated 30 more cases. On these 60 cases, TRIAGE is now on par with GPT-5.1 zero-shot on consistency, correctness, and helpfulness. The clinicians still rate dialectical reasoning as more helpful than one-sided reasoning, which is the main claim of this study. We have updated the results in Appendix F.1. We also acknowledge that the overall scale may still be limited, and we have added this limitation to Appendix A.

Dear Reviewer 25nT,

We sincerely thank you for taking the time to review our work and for providing constructive feedback.

We have carefully considered your concerns and addressed them through detailed responses and additional experiments, revising our manuscript accordingly. We would greatly appreciate it if you could revisit our rebuttal and share any further comments or outstanding concerns. We would be happy to provide additional clarification if needed.

Thank you once again for your time and consideration.

Best regards,
The Authors

Dear Reviewer 25nT,

We fully understand how demanding this period is, and we truly appreciate the time and effort you have already devoted to our submission. We are writing to gently follow up on our rebuttal, as the discussion period will close in 24 hours.

In our response, we have done our best to address each of the concerns you raised, supported by additional experiments and corresponding revisions to the manuscript.

We would be grateful if you could let us know whether our response has resolved your concerns, or whether any points remain unclear. We would be happy to respond promptly to any remaining questions within the time left.

Thank you again for your time and thoughtful feedback.

Best regards,

The Authors

Reviewer_kw4QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed TRIAGE, a LLM-based framework for EHR Risk Prediction with a calibrated risk score and interpretable rationales. The paper first analyzed the major problems with existing EHR Risk Prediction with LLMs approaches, where conventional LLMs typically overcommit to a predicted outcome before answering leading to overconfident risk scores (P1); while LLMs’ rationales are typically one-sided, where they will only present evidence for the committed outcome leading to biased rationales (P2). The paper proposed a multi-stage training framework to tackle these two issues, with two main steps: Dialectical Reasoning Supervision and On-policy Self-refinement. Dialectical Reasoning Supervision tackles P2 by training the model to provide rationale for both available outcomes through SFT. To alleviate P1, the framework instructs the model to create a decision token without any intermediate summary or verdict. Afterwards, the paper introduced further post-training with On-policy Self-refinement with GRPO for better rationale and calibrated probabilities. The final methods demonstrated impressive performance on multiple EHR Risk Prediction benchmarks in terms of both predictive accuracy and calibration.

Critical strengths
  1. Clearly defined motivations, backed up with experimentation: The authors position their research questions quite logically, with each point being backed up by preliminary experimental results. The identified problems are all critical for adopting LLMs for EHR Risk Prediction, leading to poor performance and calibration.

  2. Extensive and informative experimental results: The method is tested on five EHR benchmarks, which achieves improved results compared with previous methods. The authors also provided ablation studies on each component in the proposed frameworks, while also evaluating LLM rationales' quality and faithfulness with both LLM-as-a-judge and clinicians' evaluation.

Critical weaknesses
  1. Limited Methodological Novelty: While the paper’s motivations are clear, logical and well-analyzed, the solutions by the paper are limited in novelty. LLM's reasoning overconfidence is a well-known problem in existing LLM works [1],[2],[3]. The methodological contributions boil down to: (1) finetune the model to generate rationales for both outcomes instead of one, which is already established [4],[5]; (2) Obtaining output's probabilities from 0/1 token (already well-established); (3) batch-level pairwise margin reward GRPO reward for better cross-patient comparability is also inspired by a similar paper as the author stated.

  2. Narrow and Limited Reasoning Framework: The Dialectical Reasoning Supervision objectives were effective for binary classification tasks, however it does not generalize to other essential EHR predictive tasks such as ICD Diagnosis, an extreme multilabel classification problem with thousands of potential outcomes; or predicting continuous vital values. Also, from the results presented in Table 5, the reasoning by the model seems to have a small role in boosting the predictive performance (from 86.4 to 86.9 AUROC), only serving as an interpretability tool for clinicians. TRIAGE's reasoning also didn’t achieve any improved consistency or correctness over original one-sided reasoning.

[1] Leng, Jixuan, et al. "Taming overconfidence in llms: Reward calibration in rlhf." International Conference on Learning Representations. Vol. 2025. 2025. [2] Chhikara, Prateek. "Mind the confidence gap: Overconfidence, calibration, and distractor effects in large language models." arXiv preprint arXiv:2502.11028 (2025). [3] Damani, Mehul, et al. "Beyond binary rewards: Training lms to reason about their uncertainty." International conference on learning representations. Vol. 2026. 2026. [4] Yao, Liang, and Yang Yang. "Large language models are contrastive reasoners." Expert Systems with Applications (2025): 130407. [5] Chia, Yew Ken, et al. "Contrastive chain-of-thought prompting." arXiv preprint arXiv:2311.09277 (2023).

Questions for authors

Q1 Explanation on the role of reasoning traces: Why was TRIAGE's dialectical reasoning not better in terms of correctness or consistency than the original flawed one-sided reasoning? Isn't less bias and more accuracy in the final predictions the goal for this module? Also the conclusion from this experiment (Section F1) seems unreliable due to small sample sizes, with only 30 ICU stays and 2 clinicians. The rationale seems to contribute only a small amount to the final predictive accuracy too. Does reasoning actually influence the model to make better predictions, or is it just a post-hoc interpretability tool? A suggestion for an ablation study would be to measure probabilities shift after: deleting all rationales, deleting only the counter-evidence reasoning chunk, or swapping in a false rationale from a similar patient.

Q2 Calibration comparison with ISMTS specialized models: TRIAGE demonstrated improved calibration against existing LLMs approach, which are already established to have overconfident probabilities. But how does TRIAGE's calibration compare with specialized ISMTS models, both raw and after post-hoc recalibration?

Q3 Explanation about batch-level reward effects: Batch-level reward was proposed for better cross-patient calibration, but ended up offering little improvements in terms of calibration too (Figure 7), while improving discriminative AUPRC results. Can you provide an explanation for this phenomenon?

Q4 Sources of calibration gain: Does dialectical reasoning and GRPO have any effects on the model's calibration? Table 2 did show marginal improvements after prompting zero-shot LLMs for two-sided reasoning, but what about after TRIAGE's training? Does two-sided reasoning and GRPO offer better calibration than an answer-only model? Explain the observed results, whether positive or negative. Q5 GRPO effect on LLM Reasoning: Does GRPO help improve the LLM's reasoning quality, which are typical in LLMs, or does it only affect the final token’s prediction?

Q6 Computational tradeoff between Reasoning and no Reasoning: Since Reasoning seems to offer small improvements in terms of performance only (with better interpretability to be fair), but with many more tokens generated, so the inference time will be higher. Compare inference cost of an answer-only and reasoning model. Does the difference in inference cost justify the benefits given by the rationales?

7 comments
AuthorsOpenReview ↗

Response to Reviewer kw4Q (1/4)

We thank the reviewer for the thoughtful review and detailed feedback. We appreciate the recognition that our motivations are clearly defined and supported by preliminary experiments. We are also grateful for the acknowledgment of the extensive experiments, component ablations, and assessments of rationale quality and faithfulness by LLM judges and clinicians.


[W1] Limited Methodological Novelty

We thank the reviewer for raising this important question about the novelty of TRIAGE. Our contribution centers on a unified formulation for ISMTS: a single LLM that acts as both reasoner and risk predictor, with its generated rationales directly conditioning its own risk score (Table 1). We carefully scrutinized the challenges needed to enable this dual role. We acknowledge prior work on overconfidence and verbalized confidence [1–3], on contrastive reasoning [4,5], and on extracting token probabilities, and we do not claim any of these as new in isolation. To get a graded risk score from the LLM itself, we needed both a token-level readout and dialectical reasoning.

  1. A token-level readout requires two-sided reasoning. Extracting the token probability does not by itself give a graded score. We show in Section 4 that conventional reasoning pushes this probability toward 0 or 1 because of one-sidedness and pre-commitment. To avoid this, we fine-tune the model to write rationales for both outcomes before it scores. In [4,5], one side is wrong by construction and is used as a negative example to improve answer accuracy. In TRIAGE, both rationales cite evidence from the same trajectory, and the risk score is conditioned on both to produce a graded score that incorporates competing evidence rather than favoring either extreme.
  2. A graded risk score requires a token-level readout. In triage, the model must distinguish degrees of risk across patients, including patients with the same predicted outcome. Methods that report a confidence together with reasoning verbalize it [1–3]. However, we found that a verbalized score is coarse and gives a weak training signal (Appendix B.2). Therefore, we read the risk score from the probability of the 0/1 token, which is conditioned on the generated rationales.

Beyond these two required components, we add a third that is new as a GRPO reward.

  1. Batch-level GRPO reward. Our reward is continuous and computed from the answer token probability pp. While designing the reward, we needed a way to judge whether the given pp is appropriate for cross-patient discrimination. However, this is hard to do from one patient alone since the same pp can be too high or too low depending on what the model assigns to other patients. We therefore treat all rollouts in a batch as one group and reward each rollout by the margin between its pp and the pp assigned to patients with the opposite outcome. As a result, rollout’s reward depends on the predictions for other patients, which is not the case in standard LLM GRPO.
AuthorsOpenReview ↗

Response to Reviewer kw4Q (2/4)

[W2-1] Narrow and Limited Reasoning Framework

We agree that our experiments cover only binary prediction tasks, and we now explicitly state this in the Limitations section (Appendix A). The core step of TRIAGE is to reason about each candidate outcome before estimating its probability, and this step can be applied beyond binary tasks once the task is reformulated. Below we describe how we extend TRIAGE to the two settings the reviewer raises.

Multi-label classification. We treat each candidate diagnosis as a binary question and apply dialectical reasoning to each one independently. To keep the cost manageable over thousands of labels, we follow prior LLM-based medical coding work [6, 7] and narrow the candidates first: GPT-5.1 proposes the top-10 diagnoses, and TRIAGE evaluates each of them. On the same candidates, TRIAGE outperforms zero-shot GPT-5.1 on all three metrics.

kw4Q-Table 1. Multi-label ICD-9-CM diagnosis prediction on eICU-demo. AUROC and AUPRC treat each (sample, class) pair as a binary prediction.

MethodAUROCAUPRCMAP
zero-shot GPT-5.171.928.225.7
TRIAGE$_{\text{SFT}}$87.556.537.6

Continuous vital-sign forecasting. We discretize the target values into bins and treat forecasting as multi-class prediction, so that TRIAGE reasons about each bin before estimating the bin probabilities. LLMs are known to be strong zero-shot forecasters [8], so GPT-5.1 is already a strong baseline, and TRIAGE improves on it.

kw4Q-Table 2. Continuous vital-sign forecasting on P12: predicting mean arterial pressure (MAP) 6 hours ahead.

MethodMAERMSE
Keep Last Value11.1015.90
zero-shot GPT-5.18.9512.38
TRIAGE$_{\text{SFT}}$8.5311.79

[6] Boyle, Joseph S., et al. “Automated clinical coding using off-the-shelf large language models.” NeurIPS 2023 Workshop on Deep Generative Models for Health. 2023.

[7] Motzfeldt, Andreas, et al. “Code like humans: A multi-agent solution for medical coding.” Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.

[8] Gruver, Nate, et al. “Large language models are zero-shot time series forecasters.” Advances in Neural Information Processing Systems 36. 2023.


[W2-2, Q1-1] Contribution of Reasoning to Predictive Performance

We address the two points in turn.

  1. Interpretability tool. A clinical decision support system requires not only strong predictive performance but also interpretability. In clinical settings, it is not a supplementary tool but a necessary component that has to be pursued together with discrimination performance [9, 10]. Although answer-only SFT and ISMTS models achieve strong discrimination, their inability to explain predictions limits their deployment in clinical practice.
  2. Performance gain. The gain from reasoning is larger than the AUROC difference suggests. Because of the severe class imbalance, AUPRC is the more informative metric in our setting. With SFT alone, our reasoning model achieves an AUPRC of 56.4, compared to 53.4 for the answer-only model. Moreover, by generating both probabilities and dialectical reasoning, our model supports further optimization through GRPO, improving AUPRC to 59.0. Such RL optimization is difficult to apply to answer-only models.

[9] Amann, Julia, et al. “Explainability for artificial intelligence in healthcare: a multidisciplinary perspective.” BMC Medical Informatics and Decision Making 20.1. 2020.

[10] Tonekaboni, Sana, et al. “What clinicians want: Contextualizing explainable machine learning for clinical end use.” Machine Learning for Healthcare Conference. 2019.

AuthorsOpenReview ↗

Response to Reviewer kw4Q (3/4)

[W2-3 & Q1-2] Human Study: Dialectical vs. One-Sided Reasoning, and Sample Size

We thank the reviewer for these questions. Both concern the human study (Appendix F.1), in which clinicians rate each rationale on consistency, correctness, and helpfulness.

  • Consistency and correctness: no gap expected. These two metrics rate properties that one-sided reasoning does not fail on, so we did not expect a gap (exact criterion definitions in Figure 26). Consistency asks whether the rationale agrees with the model’s prediction and is free of contradictions. A one-sided rationale argues only for the prediction it reaches, so it is consistent almost by construction. Correctness asks whether the medical knowledge and reasoning in the rationale are valid. A one-sided rationale can still be medically correct even when it is biased, provided the facts it cites are valid. Its flaw lies in the evidence it leaves out, and neither metric penalizes omitted evidence.
  • Helpfulness and predictive performance: where dialectical reasoning helps. As the reviewer says, the goal of dialectical reasoning is less bias and more accurate final predictions. In the human study, this benefit appears in the third metric, helpfulness: clinicians rated dialectical rationales higher than one-sided ones (4.68 vs. 4.02, Table 13), because examining the plausible evidence for the opposite outcome is also helpful to them. In the predictions themselves, one-sided reasoning suffers from risk polarization and its discrimination performance collapses, whereas dialectical reasoning avoids this failure (Table 5).
  • Sample size: extended from 30 to 60 ICU stays. We agree that a broader clinical evaluation is valuable. To address this concern, we extended the study: the same two clinicians rated 30 additional cases, and the updated results are in Appendix F.1. The main change is that TRIAGE is now on par with GPT-5.1 zero-shot on all three metrics. Our main claim is unchanged: dialectical reasoning is rated more helpful than one-sided reasoning. We also acknowledge that the overall scale may still be limited, and we have added this limitation to Appendix A.

[Q1-3] Influence of Reasoning on the Prediction: Ablation Study

We thank the reviewer for this comment. Following the reviewer's suggestion, we conducted additional experiments that edit the generated reasoning at inference. The results, together with two findings already in the paper, indicate that the reasoning in TRIAGE influences the prediction and is not only a post-hoc explanation.

In the paper: (i) removing the measurements that the rationale cites changes the prediction, whereas removing random measurements has little effect (Sec. 7.3); (ii) on P12, AUPRC rises from 53.4 (answer-only) to 56.4 with reasoning supervision and to 59.0 after RL.

In the additional experiments (kw4Q-Table 3), deleting all rationales lowers AUPRC on P12 by 1.59 points (58.98 to 57.39). Removing only the rationale for the counter (incorrect) outcome, a diagnostic edit that uses the true label, raises it by 1.62 points (58.98 to 60.60). Both shifts go in the direction we expected, which suggests that the reasoning affects the prediction. The shifts are modest, likely because the features that the reasoning cites remain in the patient record after the edit.

kw4Q-Table 3. Predictive performance on P12 when the generated reasoning is edited at inference. ΔAUPRC is the difference from the unedited row.

Edit to the generated reasoningAUROCAUPRCΔAUPRC (points)
None (unedited)87.32 ± 1.2258.98 ± 3.44–
Deleting all rationales86.64 ± 1.1757.39 ± 3.12−1.59
Removing only the rationale for the incorrect outcome87.60 ± 0.8260.60 ± 3.58+1.62

[Q2] Calibration Comparison with ISMTS Models

Thank you for raising this point. Our main interest was whether the probability that an LLM produces after its reasoning is well calibrated, so Table 4 of the paper compares TRIAGE with LLM-based approaches. kw4Q-Table 4 shows the calibration performance of specialized ISMTS models, raw and after Platt scaling (averaged over five datasets). Before recalibration, TRIAGE has the lowest ECE (0.038 vs. 0.183 to 0.191 for the specialized models) and the lowest Brier score (0.077 vs. 0.136). After Platt scaling, all models reach a similar level (ECE 0.016 to 0.018, Brier 0.073 to 0.075).

kw4Q-Table 4. Calibration scores of specialized ISMTS models and TRIAGE, raw and after Platt scaling. Values are averaged over five datasets. Platt scaling is fit on the validation set.

ModelECE (raw)ECE (Platt)Brier (raw)Brier (Platt)
GRU-D0.1900.0180.1360.075
KEDGN0.1910.0160.1360.074
STraTS0.1830.0170.1360.075
TRIAGE$_{\text{SFT}}$0.1800.0170.1380.075
TRIAGE0.0380.0170.0770.073

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Rebuttal and Additional Experiments

We thank all reviewers for their thoughtful comments and constructive feedback, which have substantially strengthened our work. We are encouraged that the reviewers found that TRIAGE addresses an important clinical prediction problem (pvxz) with clearly defined motivations supported by preliminary experiments (kw4Q). We appreciate the reviewers' acknowledgement of our identification of the risk polarization problem (pvxz, 25nT) and of the verdict-free two-sided reasoning as a potentially transferable insight (25nT). Additionally, we value the recognition of our extensive evaluation across five clinical benchmarks (pvxz, kw4Q), including ablations and assessments of rationale quality and faithfulness (kw4Q).

We have provided detailed responses to each reviewer's comments in the individual threads. Below we summarize our response on methodological novelty and the additional experiments from the rebuttal.

Methodological novelty. Reviewers kw4Q and pvxz ask how TRIAGE differs from prior techniques. Clinical risk prediction requires both an explanation and a risk score that ranks patients for triage, and existing LLM-based methods for ISMTS do not provide the two together (Table 1). Our contribution centers on a formulation in which a single LLM acts as both reasoner and risk predictor: it writes a rationale and then outputs a graded risk score conditioned on that rationale. The techniques it uses (two-sided rationales, the answer-token probability, and GRPO) each have counterparts in other work. TRIAGE does not combine them for their own sake: each addresses a need of this formulation, as the two components below show.

  • A reasoning structure that keeps the probability graded. We identify risk polarization: when an LLM reasons toward one outcome before it answers, its probability is pushed toward 0 or 1 and it ranks patients poorly (Section 4). TRIAGE mitigates this by reasoning about both outcomes and omitting any concluding verdict. Reviewer 25nT notes that this design makes sense in context and is supported by our analysis. The additional comparisons (pvxz-Table 11) show a similar collapse in an existing reasoning-based clinical method.
  • A batch-level reward for cross-patient ranking. This is our methodological addition on the training side. In standard LLM GRPO, each rollout is rewarded on its own. Our reward depends on the probabilities assigned to patients with the opposite outcome in the same batch, so it directly rewards ranking patients against each other (Section 5.2).

In response to the reviewers' valuable comments, we conducted the following additional empirical evaluations and analyses during the rebuttal:

  • Controlled comparisons. Controlled ablations confirm that both dialectical reasoning and the batch-level pairwise margin reward contribute to performance.
  • Calibration and clinical baselines. Compared with specialized ISMTS models, TRIAGE achieves the lowest raw calibration error (ECE 0.038 vs. 0.183-0.191) and remains comparable after Platt scaling. We also compared TRIAGE with severity scores used in clinical practice (SOFA, SAPS I/II, and APACHE II) on P12, where it attains higher sensitivity at matched specificity (25nT-Table 1).
  • Explanation quality and faithfulness. We extended the clinician human study to 60 cases with an additional explanation baseline; dialectical rationales are rated more helpful than one-sided ones and on par with GPT-5.1. Extended deletion experiments further show that the measurements cited in TRIAGE's rationales are more necessary and more sufficient for its risk score than those from alternative explainers.
  • Out-of-domain generalization. Without any target-domain training, TRIAGE generally transfers better across datasets than Record2Vec. Under held-out-hospital and temporal validation, it remains comparable to specialized ISMTS models even at the SFT stage.
  • Expansion of task scope. Preliminary extensions to multi-label diagnosis prediction and vital-sign forecasting indicate that TRIAGE can be applied beyond binary tasks.

Finally, we further clarified several points raised by the reviewers, including the handling of irregular sampling, the presentation of specialized ISMTS models, the P12 results in Appendix G.1, inference cost, and checkpoint selection. The manuscript has been revised accordingly, and the remaining results will be included in the final version.

We are grateful to the reviewers and ACs for their time and effort, which have helped improve our work. We hope that our rebuttal, additional experiments, and revised manuscript address the reviewers' concerns and demonstrate the effectiveness of TRIAGE for explainable clinical risk prediction with a single LLM.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.