acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Transparent Reasoning through Aligned Concept Explanations in Clinical Reinforcement Learning

Abstract

When an offline reinforcement learning (RL) policy recommends a treatment that differs from observed clinical practice, the disagreement alone cannot reveal whether the departure reflects clinically useful personalization or limitations in the data or learning objective. We introduce Transparent Reasoning through Aligned Concept Explanations (TRACE), a framework that explains and directly compares a learned RL policy with an estimated behavior policy fitted to recorded treatment actions. TRACE uses a shared encoder and sparse autoencoder to define a common concept space, then conditions a shared LLM-based explainer on the patient state and policy-relevant concepts while withholding the selected action. Teacher supervision and proximal policy optimization promote hidden-action recovery, concept consistency, state grounding, and resistance to action leakage. We evaluate TRACE on three MIMIC-III/IV tasks, including renal replacement therapy, noninvasive ventilation, and vasopressin initiation, where the policies disagree on 11.9%–33.4% of held-out states. TRACE achieves the highest hidden-action recovery accuracy and macro-F1 among the evaluated explanation methods for both policies. On disagreement states, each policy's action is recovered more accurately from its own explanation than from the other policy's explanation. Concept perturbations reduce learned-policy recovery across all tasks, while policy-side interventions show that cited concepts are more influential than uncited or random concepts, most clearly for RRT. These results show that TRACE enables faithful, concept-grounded comparison of learned and behavior policies, providing a basis for pre-deployment auditing and targeted clinical review of disagreements.

Primary area:
interpretability and explainable AI
Keywords:
Reinforcement Learning, Policy Explanation, Faithful Explanations, Concept-Based Interpretability, Sparse Autoencoders, Policy Comparison

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TaVoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates how to create faithful natural language explanations for differences in action selection between two RL policies. The paper proposes TRACE, a framework for training an LLM that produces such explanations. For two RL policies using the same state encoder, TRACE learns a concept dictionary using top-k SAE on the encoder. The dictionary maps each feature in the SAE to some interpretation. Each feature is interpreted by comparing the training subjects across different activation values of that feature. The explainer LLM is initialized from Llama-3.1-8B-Instruct, SFT’d on teacher explanations, then PPO’d. For SFT, the teacher explanations are generated by GPT5.6-terra, using the patient state and selected concepts as input. For PPO, a frozen reader LLM scores whether the explanation helps recover the action. The paper compared TRACE against naive baselines and other existing policy-explanation methods, including methods with and without additional training. TRACE attains the highest F1 score for action prediction across all three tasks and both policies.

Critical strengths

The question that the paper tries to address is an interesting one. As more clinical ML systems are deployed, it is crucial for clinicians to understand the reasoning behind the recommended action from the RL policy, when it diverges from the population policy.

Critical weaknesses
  • In order to build the shared concept dictionary, TRACE requires the policies to use the same encoder. This in turn necessitates the training the encoder and policies from scratch. Also the framework only targets RL policies. Both points limit the general applicability of the framework.

  • The paper used general purpose LLMs for the explainer and the reader, as well as baseline in the evaluations. Given that we are in the clinical domain, it is important that we also test the effectiveness of the framework on specialized clinical LLMs (e.g. medgemma), and against baseline methods with these clinical LLMs.

Questions for authors
  • On L392-393, how is it determined whether an explainer cites a concept? On L410-411, how is “policy-selected cards” defined?

  • Across Table 1 and 2, it seems like the RL policy is noticeably more difficult to explain than the base policy. Do you have a hypothesis for why that might be the case?

  • To address the second applicability issue, instead of training the state encoder, have you tried using a pretrained medical sequence encoder (e.g. CLMBR-T)?

  • Right now the reader is about as capable as the explainer, and there is significant gap between the reader LLM and the judges used for evaluation. Have you tried varying the capability of the reader, relative to the explainer?

4 comments
AuthorsOpenReview ↗

Response to Reviewer TaVo

We thank the reviewer for the careful reading and constructive feedback. We have uploaded a revised PDF with changes highlighted in blue. Below, we address each point raised in the Weaknesses and Questions sections.

W1.

TRACE requires a shared encoder to place both policies in a common representation space, enabling the SAE to construct a unified concept dictionary for comparing their decisions. However, this does not require training the encoder or policies from scratch. The encoder can be pretrained and frozen, with policy heads trained independently. Moreover, TRACE is not restricted to RL policies: our behavior policy is trained through supervised learning using an adapter and linear head optimized with cross-entropy loss. Only the RL-specific concept ranking relies on Q-values and the support rule.

We have clarified these requirements in Section 2.2 and introduced three encoder variants (Q3) demonstrating that the shared representation need not be learned jointly (an RL-trained encoder with a behavior head fitted afterward), trained through RL (a self-supervised encoder learned without policy objectives), or trained from scratch (BAT, a frozen ICU encoder pretrained on eICU and MIMIC-III). We also acknowledge that TRACE does not directly support policies without an accessible, compatible representation, such as gradient-boosted trees, and have added this limitation to the paper.

W2 (Clinical LLMs).

We thank the reviewer for this suggestion. We conducted additional experiments using MedGemma-4B and MedGemma-27B [1] as explainers on the RRT test set, comparing them with Llama-3.1-8B under both zero-shot baselines and TRACE. The results report action recovery accuracy (RL / behavior):

BackboneDirect LLMZero-shot CoTTRACE
Llama-3.1-8B0.452/0.4950.438/0.4570.755/0.935
MedGemma-4B0.422/0.3420.412/0.3700.697/0.912
MedGemma-27B0.400/0.3770.443/0.395–

At the standard token limit, neither MedGemma model surpasses the no-explanation judge baseline (0.453/0.488) in the zero-shot settings, suggesting that clinical knowledge alone is insufficient to recover policy-specific decisions. However, after training with the TRACE pipeline using a 27B reader, MedGemma-4B achieves 0.697/0.912, approaching Llama-3.1-8B (0.755/0.935), with gaps of only 0.058/0.023. These results demonstrate that TRACE can effectively extend to specialized clinical LLMs, with substantial improvements over their zero-shot baselines. Due to computational constraints, we did not conduct TRACE post-training for MedGemma-27B.

Q1.

We identify cited concepts with a rule-based text matcher that uses no LLM or other learned model. Each concept card specifies clinical variables and their directions (e.g., higher creatinine with lower urine output); the matcher finds these in the explanation through a predefined list of aliases and attributes each matching statement to the card that asserts it. A card counts as cited if its name is explicitly mentioned or a consecutive run of directional statements matches it. Against the card use the teacher recorded (19,096 RRT records) it has precision 0.936. Cp(s)C_p(s) is the set of active concepts with the highest score combining contribution to the representation, effect on the policy's action scores and effect on the margin between the selected action and the runner-up, with a redundancy penalty. Both definitions are now in the main text.

Q2 (Why RL is harder to explain).

We appreciate this insightful question. While we do not yet have a definitive explanation, we hypothesize that several factors may contribute to this gap.

First, the behavior policy closely reflects observed clinical practice, which may align better with the prior clinical knowledge of pretrained LLMs. Second, RL decisions appear more complex and less predictable from the state alone: on RRT, logistic regression recovers behavior actions with 0.901 accuracy, compared with 0.819 for RL actions, and the judge without an explanation shows a similar pattern across all three cohorts. Third, RL decisions depend on both Q-values and the support rule, with relatively small action margins (median Q-margin 0.14 versus Q-value SD 1.16).

AuthorsOpenReview ↗

Response to Reviewer TaVo (continued)

Q3 (pretrained encoder).

We thank the reviewer for the suggestion. However, CLMBR-T [2] reads coded Stanford timelines through its own vocabulary; mapping MIMIC's numeric vitals and labs into it is not supported by public tooling and would change the policies. We used a frozen pretrained ICU encoder that matches our data, BAT [3, 4] (pretrained on eICU and MIMIC-III), plus two controls, and ran TRACE's representation stages unchanged on RRT:

paperRL-only encoderself-supervisedfrozen BAT
actions kept by SAE, RL/beh. (%)99.1/99.699.0/99.898.5/99.197.4/98.1
named features replicating (%)98.197.796.196.1
RL flip, 8 selected vs 8 random (%)39.5 vs 18.625.1 vs 14.815.0 vs 6.530.1 vs 12.8

The selected cards stay decision-relevant on every encoder, so TRACE needs a shared representation, not a particular training recipe. The BAT policy reads 24 h of history and is a different policy (63% action agreement), and we did not retrain the explainer on these variants.

Q4 (reader capability).

The paper's reader is Qwen3-8B: it scores every sentence of every rollout, so it must be small; gains are measured by a judge PPO never saw. Seven readers (Qwen3-1.7B to 32B, MedGemma-4B/27B) re-scored the frozen RRT explanations. The best (32B) recovers 0.538 / 0.533 of RL / behavior actions against 0.755 / 0.935 for Opus 5. Yet where the policies differ (119 states), the matching explanation beats a transplanted one on the RL target (0.571 vs 0.277 at 8B). Moreover, two independent judges, Opus 5 and GPT-6 Astra, exhibit substantial state-level disagreement (preference correlation 0.15) despite broadly consistent aggregate results. Such disagreement is not unexpected, as the evaluation judges serve as proxies for clinical experts, who may also differ in their assessments of individual cases. Nevertheless, this highlights the importance of accounting for judge specific biases, which we recognize as an important direction for future work.

W2: General-purpose vs. clinical LLM explainers on RRT (600 test states; same evaluation judge).

ExplainerMethodRL Acc. / F1Behavior Acc. / F1
NoneNo explanation0.453 / –0.488 / –
Llama-3.1-8BDirect LLM0.452 / 0.2930.495 / 0.373
Llama-3.1-8BZero-shot CoT0.438 / 0.2910.457 / 0.382
MedGemma-4BDirect LLM0.422 / 0.3510.342 / 0.321
MedGemma-4BZero-shot CoT0.412 / 0.3500.370 / 0.350
MedGemma-27BDirect LLM0.400 / 0.3450.377 / 0.358
MedGemma-27BZero-shot CoT0.443 / 0.3580.395 / 0.346
MedGemma-27BCoT (2,048 tokens)0.463 / 0.3650.397 / 0.421
Llama-3.1-8BTRACE w/o PPO0.742 / 0.6720.897 / 0.908
Llama-3.1-8BTRACE (8B reader)0.755 / 0.6880.935 / 0.926
MedGemma-4BTRACE w/o PPO0.690 / 0.6260.837 / 0.857
MedGemma-4BTRACE (27B reader)0.697 / 0.6300.912 / 0.899

Specialized clinical LLMs do not consistently outperform general-purpose models in zero-shot settings. Increasing MedGemma-27B's generation limit to 2,048 tokens yields only modest improvements, with RL accuracy (0.463) remaining close to the no-explanation baseline (0.453). In contrast, TRACE substantially improves MedGemma-4B's performance, achieving 0.697/0.912 RL/behavior accuracy. PPO further improves its behavior accuracy by +0.075 (95% CI: [+0.041, +0.112]), while the RL gain is smaller (+0.007, 95% CI: [−0.035, +0.052]). These findings suggest that policy-specific training is more important than clinical specialization alone for recovering policy decisions.

References

[1] Sellergren, A., et al., 2025. MedGemma technical report. arXiv preprint arXiv:2507.05201.

[2] Wornow, M., Thapa, R., Steinberg, E., Fries, J. and Shah, N., 2023. EHRSHOT: An EHR benchmark for few-shot evaluation of foundation models. Advances in Neural Information Processing Systems, 36.

[3] DeVries, R., Christensen, C., Zepeda Mendoza, M.L. and Winther, O., 2025. Bi-axial transformers: Addressing the increasing complexity of EHR classification. arXiv preprint arXiv:2508.12418.

[4] Jagd, K.N., DeVries, R. and Winther, O., 2025. Towards self-supervised foundation models for critical care time series. arXiv preprint arXiv:2509.19885 (NeurIPS 2025 Workshop on Learning from Time Series for Health).

Reviewer_TaVoOpenReview ↗

Q2 response is speculative. Keeping score

I'd like to thank the authors for the extensive responses to the points in my review. However, the response to Q2 remains speculative without empirical attribution or failure-mode analysis. In particular, showing lower action predictability from the state (e.g., via logistic regression) conflates policy complexity with post-hoc explainability, leaving open why generated explanations fail when the RL policy diverges from standard clinical practice. I'm going to keep my score.

We thank the reviewer for the follow-up. We agree that lower action predictability does not necessarily imply poorer explainability. To investigate the actual failure modes, we conducted an additional analysis on 600 RRT test states.

What causes lower RL explanation recovery? Without explanations, the judge predicts RL and behavior actions at similar rates (0.453 vs. 0.488). With TRACE explanations, recovery reaches 0.755 for RL and 0.935 for behavior, suggesting that state-level policy complexity alone cannot explain the gap.

Our analysis identifies two main sources of RL explanation failures:

  1. Ambiguous action values: When supported actions have similar Q-values, the explanation may not clearly distinguish the selected action from its closest alternative. RL recovery increases from 0.546 to 0.794 between low- and high-Q-margin states.

2. Behavior-support constraints: BCQ may exclude a higher-Q action because it falls outside the clinician-supported action set. Since explanations focus on clinical evidence rather than these constraints, the judge may favor the excluded action. As future work, we will investigate whether this limitation persists across different RL algorithms.

Together, these two failure modes account for 130 of 147 (88%) RL recovery errors. Importantly, 112 of these 147 errors occur when RL and behavior policies agree. Thus, the lower RL recovery is primarily associated with ambiguous action values and support constraints, rather than disagreement with clinical practice itself.

We hope this empirical analysis addresses the reviewer's concern.

Reviewer_6MsGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework that uses a shared encoder, sparse autoencoder, and LLM to generate comparable natural-language explanations for an offline RL policy and a behavior policy. It evaluates the framework on three MIMIC clinical tasks and finds that it improves hidden-action recovery and policy-specific explanation generation. However, because concept selection uses action-related information and evaluation relies mainly on LLM judges, the results do not fully establish that the explanations faithfully reflect the policies’ internal decision processes.

Critical strengths

1.Clear and important research problem. The paper addresses a well-defined and meaningful problem, particularly in the context of clinical decision-making, where interpretability and reliability are critical.

2.Comprehensive experimental evaluation. The experimental design is relatively thorough, including comparisons with multiple baselines, ablation studies.

3.Clear presentation. The paper is well-organized and clearly written, making the motivation, methodology, and experimental results easy to follow.

Critical weaknesses

1.The action-blind claim is valid for the inference-time Explanation LLM interface, but does not apply to the entire explanation pipeline. TRACE uses the target action upstream for policy-specific concept selection, teacher supervision, and PPO reward computation. Therefore, the method should be interpreted as an action-conditioned evidence-selection pipeline followed by action-blind verbalization. The key remaining question is whether the resulting natural-language explanation is faithful to the policy’s internal computation, rather than merely simulating the target action from action-informed evidence.

2.The paper's primary evaluation metrics focus on simulatability rather than directly assessing faithfulness. Although the authors provide additional evidence of decision relevance through concept interventions, the effectiveness of these interventions varies across tasks. Therefore, the current experimental evidence is insufficient to fully support the paper's broader claims of faithful explanations.

3.Limited external validation. All three clinical tasks are constructed from the MIMIC datasets, which limits the external validity and generalizability of the findings.

Questions for authors

1.The action-blind claim is valid for the inference-time Explanation LLM interface, but does not apply to the entire explanation pipeline. TRACE uses the target action upstream for policy-specific concept selection, teacher supervision, and PPO reward computation. Therefore, the method should be interpreted as an action-conditioned evidence-selection pipeline followed by action-blind verbalization. The key remaining question is whether the resulting natural-language explanation is faithful to the policy’s internal computation, rather than merely simulating the target action from action-informed evidence.

2.The paper's primary evaluation metrics focus on simulatability rather than directly assessing faithfulness. Although the authors provide additional evidence of decision relevance through concept interventions, the effectiveness of these interventions varies across tasks. Therefore, the current experimental evidence is insufficient to fully support the paper's broader claims of faithful explanations.

3.Limited external validation. All three clinical tasks are constructed from the MIMIC datasets, which limits the external validity and generalizability of the findings.

4.How much does TRACE's performance degrade when the concept selector does not have access to the target action or action-conditioned decision information? An action-agnostic concept selection ablation would help clarify the contribution of action-conditioned concept selection to the overall performance.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6MsG

We thank the reviewer for the positive assessment of the problem. We uploaded a revised PDF with all changes highlighted in blue. Below, we address each point raised in the Weaknesses and Questions sections.

W1/Q1.

We agree that action-blindness applies to the explainer's inference interface, not the entire pipeline, and have clarified this distinction in Section 2.3. The concept dictionary is learned without actions, while policy-specific concept selection partially relies on an action-conditioned margin, and teacher supervision and PPO rewards use target actions. This conditioning is intentional and follows the principle of class-specific attribution: identifying evidence for a particular decision requires knowing which action is being explained. Importantly, the attribution is computed from the frozen policy, while the explainer receives neither the target action nor policy scores at inference.

To further assess whether the selected concepts reflect the policy's internal computation rather than merely support action prediction, we intervene directly on the frozen representation. Removing cited concepts changes the RRT RL action in 61.3% of disagreement states, compared with 40.2% for uncited concepts and 28.4% for random concepts. These findings provide direct evidence of the selected concepts' relevance to the policy's decisions.

W2/Q2.

We agree that decision recovery primarily measures simulatability rather than faithfulness, and have revised Section 3.1 accordingly. To provide stronger evidence of decision relevance, we conducted additional interventions on the frozen policy representation. After controlling for perturbation magnitude, removing cited concepts on RRT disagreement states still changes the RL action significantly more often than removing size-matched uncited concepts (+0.222, 95% CI [0.112, 0.337]) or random concepts (+0.331). We also repeated the test with three alternative encoders: one trained with the RL objective alone, one self-supervised without any policy objective, and one frozen ICU encoder pretrained on other databases. On all three, removing the selected concepts changes the decision more often than removing unselected concepts of matched strength: 1.2–2.1 times as often for the RL policy and 4.0–6.7 times for the behavior policy.

W3/Q3 (External validation).

We thank the reviewer for highlighting the importance of external validation. We evaluated TRACE on eICU-CRD [1], covering 208 hospitals, using the same cohort definitions as in the paper (RRT: 4,563 stays; NIV: 2,894; vasopressin: 1,730). All models were transferred directly from MIMIC without retraining or fine-tuning, including the baselines from Table 1.

RL / BehaviorMIMICeICUBest baseline (eICU)No explanation (eICU)
RRT0.755 / 0.9350.743 / 0.9280.628 / 0.8100.407 / 0.472
NIV0.842 / 0.8920.785 / 0.9150.582 / 0.6980.550 / 0.663
Vasopressin0.807 / 0.9750.828 / 0.9420.778 / 0.9670.697 / 0.728

The results show that TRACE maintains relatively stable performance across datasets, with recovery changing by no more than 5.7 percentage points. TRACE outperforms the strongest baseline in five of six settings, with the only exception being vasopressin behavior, where SFT achieves 0.967 versus TRACE's 0.942 on a highly imbalanced cohort (only 6 of 600 evaluated eICU states initiate vasopressin). Although some cohort-specific differences remain, these results provide encouraging evidence of TRACE's cross-dataset generalizability without additional training.

Q4 (Action-agnostic analysis).

We thank the reviewer for this helpful suggestion. We conducted the requested ablation on RRT, including retraining the explainer with action-agnostic concept selection.

RRTRLBehavior
Paper selector0.7550.935
Without margin term0.6230.868
Action-agnostic0.6820.820
Action-agnostic (retrained)0.5930.792
No explanation0.4530.488

The observed performance decrease is consistent with the intended role of our selector, which identifies concepts relevant to the policy's selected action. Removing action-conditioned information naturally makes it more challenging to identify the most decision-relevant evidence. Specifically, RL recovery decreases by 7.3 percentage points under action-agnostic selection and by 16.2 points after retraining. Nevertheless, both settings outperform the no-explanation baseline, suggesting that action-agnostic concepts still capture useful policy information. We appreciate the reviewer for motivating this analysis, and believe that developing a fully action-blind explanation pipeline is an interesting direction for future work.

Reference: [1] Pollard, Tom, et al. "eICU Collaborative Research Database" (version 2.0). PhysioNet (2019). RRID:SCR_007345. https://doi.org/10.13026/C2WM1R

Reviewer_vhZ2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

At its core, this paper proposes a new explanation method for RL policies called TRACE. It motivates RL-policy explanation in clinical contexts, where proposed actions by two policies may differ, and clinicians may benefit from explanations describing the policies. The paper compares TRACE with other RL-explanation methods on three clinical tasks, and shows that (i) policy actions are best recovered from TRACE explanations and (ii) perturbing the patient state and concepts changes TRACE's explanation.

Critical strengths

Important problem. Researchers increasingly propose clinical RL policies to help with treatment decisions. The recommendations of these policies are often different from the actions of clinicians. However, a clinician cannot act on a policy that she does not understand. The paper gives a good motivation for this problem, i.e. of explaining RL policies. It uses three realistic ICU tasks (RRT, NIV, vasopressin) from MIMIC-III/IV.

Comprehensive faithfulness evaluation. The paper adds faithfulness experiments that check whether an explanation really reflects the policy. On the RRT task, removal of concepts that an explanation cites changes the RL action in 61.3% of states. Removal of the same number of other concepts changes the action in only 40.2% of states. Thus, the extracted concepts have a significant effect on the policy decision and the explanations are faithful.

Critical weaknesses

Please refer to questions.

Questions for authors
  • Unclear writing. for instance, Line 186 has a circular definition of a encoder as h(s) = E(s) \in R^d, with no definition of E anywhere in the paper. There are many instances of this, such as "FQE" which is not defined anywhere but mentioned in Fig.1, but I will not belabor this point. The paper is clearly fully LLM-written, which is fine on the surface, but it makes it impossible to understand the central argument.

  • Weak motivation. The paper wishes to compare 2 policies: a "learned" policy from RL and a "behavior" policy which simply mimics the clinician. The learned policy is trained with offline RL on clinical rewards, and the behavior policy with cross-entropy on clinician actions. But the clinical rewards are arbitrary (hand-weighted combinations of mortality, SOFA, lactate etc., different per task). In the clinical setting, such rewards are not verifiable due to lack of counterfactuals, so to my mind, the optimal learned policy is a behavior policy i.e. to copy the clinician. Thus, I cannot understand why difference in actions between the two policies are clinically relevant and deserve explanations.

  • Weak evaluation. I find three issues with the evaluation. First: the motivation is to explain differing policies, but TRACE explains only one policy at a time, i.e. it is a generic RL policy explainer. I don't see how measuring whether policy actions can be recovered from natural-language explanations tackles the motivation of comparing two RL policies. Second, there already exist several explainable RL-baselines such as Gajcin 2021 and Amitai which the paper cites but are not evaluated as baselines. Finally, the faithfulness metric in Table 1 is precisely TRACE's training objective, whereas baselines have different training objectives; thus, any outperformance is trivially expected and not interesting.

  • Limited novelty. TRACE is an incremental combination of RL explanation (Yang 2026) and bottlenecked concepts. Can you clarify what the novelty is here?

  • (minor) Please show confidence intervals for Table 1.

5 comments
AuthorsOpenReview ↗

Response to Reviewer vhZ2

We appreciate the reviewer's recognition of the clinical motivation and faithfulness experiments results. First, we would like to clarify the two related concerns.

Motivation. We appreciate the reviewer's recognition of the importance of explaining RL recommendations that differ from clinicians' decisions. We agree that hand-designed rewards and unverifiable counterfactuals are limitations of clinical offline RL. However, these limitations highlight the need to examine learned policies rather than assuming clinician imitation is always optimal. Our learned policy uses BCQ [1], a widely adopted clinical offline RL method [2–4], which selects actions based on estimated value while remaining within the support of observed clinical practice. It differs from the behavior policy on 19.0%/11.9%/33.4% of held-out RRT/NIV/OVISS decisions (Section 3.3, Table S5). Moreover, clinical practice is not always settled: KDIGO [5] considers continuous and intermittent dialysis complementary therapies, while hospital vasopressin use ranges from 0% to 70% [6]. In OVISS [7], earlier vasopressin initiation under an RL-derived rule was associated with lower mortality in 10,217 external patients (OR 0.81). We do not assume RL decisions are necessarily superior. Rather, TRACE makes policy disagreements interpretable, helping clinicians assess whether they reflect meaningful clinical evidence or potential reward artifacts. The recorded clinician action matched the RL action in 17%/36% of RRT/NIV disagreement states.

Faithfulness: The review recognizes the concept removal results as evidence of faithfulness but raises concerns that the evaluation overlaps with our training objective. Importantly, concept removal is never optimized during training. Our new size-matched control further shows that removing cited concepts changes the RL action in 62% of RRT disagreement states, compared with 39% for equally many uncited concepts (difference: 0.222, 95% CI [0.112, 0.337]). To avoid confusion, Section 3.1 now refers to Table 1 as decision recovery, reserving faithfulness for the independent concept-removal evaluations.

AuthorsOpenReview ↗

Response to Reviewer vhZ2 (continued)

Other points in Questions

Unclear Writing. We thank the reviewer for highlighting these issues. Although we introduced EE as the encoder before the equation, we acknowledge that its definition was insufficiently clear and that FQE was not defined at first use. We have addressed both in the revision. Section 2.2 now explicitly defines EE as a three-layer ReLU network (width 64) that maps the patient state to h(s)∈R64h(s)\in\mathbb{R}^{64}. The encoder is jointly trained with both policy heads and then frozen. We also define FQE (fitted Q evaluation [8]) at first use and provide its methodology and results in the new Supplementary S7.

We would also respectfully clarify that the manuscript was written by the authors, and we take responsibility for these writing oversights. We appreciate the opportunity to improve its clarity.

Weak Evaluation.

  1. Policy Comparison. TRACE is specifically designed to compare two policies rather than explain them independently. Both policies share a unified concept dictionary (Section 2.3), and a single explainer is jointly trained on samples from both policies, including during PPO post-training. Their concept contributions are then compared at the same patient state (Section 2.4). While decision recovery evaluates one policy at a time, TRACE also enables direct comparison of their explanations. To demonstrate this, our paired-explanation evaluation (Section 3.3, Figure 3) has two independent LLM judges (Claude Opus 5 and GPT-6 Astra) compare paired explanations while blinded to their policy sources. Across 119 RRT disagreement states, Claude Opus 5 favored the RL explanation in 74 cases versus 32 for the behavior explanation, while GPT-6 Astra favored RL in 70 versus 49 cases. These results demonstrate that TRACE supports direct, side-by-side comparison of policy explanations, beyond explaining individual decisions.

  2. Additional baselines. Gajcin et al. and Amitai et al. require simulator rollouts from disagreement states, which are unavailable in retrospective ICU data, and primarily summarize outcomes rather than explain individual decisions. We therefore compare against applicable offline methods, including Yang et al. [9], policy distillation, and BC-LLM.

  3. Metric versus training objective. Although decision recovery aligns with our training objective, Table 1 is evaluated using Claude Opus 5, which is never involved in training; PPO instead uses a separate Qwen3-8B reader. Furthermore, with the same training reward, removing concept evidence reduces RL decision recovery by 14 and 31 percentage points on RRT and NIV, respectively. TRACE without PPO also outperforms all baselines in RL decision recovery, suggesting that the gains cannot be attributed solely to optimizing the recovery reward.

Limited Novelty. We agree that TRACE builds on prior work in RL explanations and interpretable concepts. Its contribution, however, goes beyond combining these components. Unlike conventional concept bottlenecks, TRACE keeps both policies fixed, requires no concept annotations, and learns concepts from their shared representation while preserving over 98% decision agreement. Its key contributions are a unified concept vocabulary for comparing two policies, a single action-blind explainer jointly trained on both, and policy-grounded evaluations through explanation swapping and concept removal. TRACE improves decision recovery over Yang et al. [9] by 28.0/32.5/3.3 percentage points across RRT/NIV/OVISS.

Confidence Intervals. We have added patient-clustered 95% confidence intervals for all entries in Tables 1–2 in the revised manuscript.

References

[1] Fujimoto, S., Meger, D. and Precup, D., 2019. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning (pp. 2052-2062). PMLR.

[2] Killian, T.W., Zhang, H., Subramanian, J., Fatemi, M. and Ghassemi, M., 2020. An empirical study of representation learning for reinforcement learning in healthcare. In Machine Learning for Health (pp. 139-160). PMLR.

[3] Fatemi, M., Wu, M., Petch, J., Nelson, W., Connolly, S.J., Benz, A., Carnicelli, A. and Ghassemi, M., 2022. Semi-Markov offline reinforcement learning for healthcare. In Conference on Health, Inference, and Learning (pp. 119-137). PMLR.

[4] Tang, S., Makar, M., Sjoding, M., Doshi-Velez, F. and Wiens, J., 2022. Leveraging factored action spaces for efficient offline reinforcement learning in healthcare. Advances in Neural Information Processing Systems, 35.

[5] Kidney Disease: Improving Global Outcomes (KDIGO) Acute Kidney Injury Work Group, 2012. KDIGO clinical practice guideline for acute kidney injury. Kidney International Supplements, 2, pp.1-138.

[6] Vail, E.A., et al., 2016. Epidemiology of vasopressin use for adults with septic shock. Annals of the American Thoracic Society, 13, pp.1760-1767.

[7] Kalimouttou, A., Kennedy, J.N., Feng, J., Singh, H., Saria, S., Angus, D.C., Seymour, C.W. and Pirracchio, R., 2025. Optimal vasopressin initiation in septic shock: the OVISS reinforcement learning study. JAMA, 333(19), pp.1688-1698.

[8] Le, H., Voloshin, C. and Yue, Y., 2019. Batch policy learning under constraints. In International Conference on Machine Learning (pp. 3703-3712). PMLR.

[9] Yang, X., Zeng, L., Dong, H., Yu, C., Wu, X., Yang, H., Wang, Y., Tambe, M. and Wang, T., 2026. Translate policy to language: Flow matching generated rewards for LLM explanations. In International Conference on Learning Representations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Major Contributions, Strengths, and Responses to Concerns

We sincerely thank the AC, SAC, PCs, and reviewers for their time and feedback. We have submitted point-by-point responses and a revised manuscript with changes highlighted in blue. Below, we briefly summarize our contributions, acknowledged strengths, and key rebuttal updates to facilitate the final assessment.

1. Core Contributions

  • Why clinical RL differs from observed practice. A central challenge in clinical RL is understanding why a learned policy recommends a different treatment from observed clinical practice. Such disagreements may reflect meaningful personalization or limitations in the data and learning objective, yet existing methods primarily explain policies in isolation or identify where their actions differ. TRACE directly addresses this gap by explaining what drives both policies' decisions for the same patient, revealing how they rely on different clinical evidence. We demonstrate this across three critical-care tasks: RRT, NIV, and vasopressin initiation.
  • Shared concepts without labels. TRACE learns a sparse concept dictionary from the representation shared by both frozen policies, enabling direct comparison of their decision-relevant evidence using a common interpretable vocabulary. No concept labels are required, and the learned representation preserves over 98% of both policies' decisions.
  • One action-blind explainer. A single LLM per cohort is jointly trained to explain both the learned RL policy and the estimated clinician-behavior policy using policy-specific concept evidence, without receiving the selected action or action scores at inference time.
  • Multiple tests of decision relevance. Independent action recovery, cross-policy swapping, frozen-policy concept interventions, and blinded two-judge comparisons probe policy specificity and grounding. TRACE improves RL action recovery over Yang et al. (ICLR 2026) by 28.0/32.5/3.4 points on the three tasks.

2. Strengths Acknowledged by Reviewers

All three reviewers recognized the importance of this clinical problem. vhZ2 and 6MsG praised the breadth of evaluation, including concept interventions, baselines, and ablations; vhZ2 explicitly acknowledged that cited concepts affect policy decisions. TaVo noted the leading action-recovery F1 across all three tasks and both policies. TaVo reported no substantive soundness concerns, while 6MsG described theirs as minor or fixable.

3. Summary of Rebuttal and Reviewer Feedback

ReviewerConcern / RequestOur Response
vhZ2Clinical motivation and cross-policy comparisonClarified that TRACE audits policy disagreements rather than assuming RL superiority. On RRT disagreement states, RL recovery is 0.706 with its own explanation versus 0.101 with the behavior-policy explanation, demonstrating policy specificity.
6MsG, vhZ2Faithfulness vs. simulatability; metric overlapClarified the distinction and used an independent judge. Policy interventions show cited concepts are more decision-relevant than matched uncited concepts (+0.222, 95% CI [0.112, 0.337]), independently of the training objective.
6MsGAction-blindness and action-agnostic selectionClarified that action-blindness applies to the explainer, not upstream selection. The requested retraining ablation achieves 0.593 RL recovery versus 0.755 originally, remaining above the no-explanation baseline (0.453).
TaVoShared encoder limits applicabilityTested three alternative encoders (RL-only, self-supervised, frozen pretrained ICU). Selected concepts remain decision-relevant without jointly training both policies.
TaVoClinical LLMs and reader robustnessEvaluated MedGemma-4B/27B and seven readers. MedGemma-4B trained with TRACE pipeline achieves 0.697/0.912 RL/behavior recovery, approaching Llama-3.1-8B (0.755/0.935).
6MsGLack of external validationConducted zero-shot validation on eICU (208 hospitals) across three cohorts. Recovery changes by at most 5.7 points, outperforming the strongest baseline in 5/6 settings.
TaVo, vhZ2Definitions, baselines, and statistical reportingClarified the encoder, concept citation/selection, FQE, and baseline applicability; added patient-clustered 95% CIs for Tables 1–2.

All results are documented in our OpenReview responses, with relevant updates in the revised manuscript. We believe these additions strengthen TRACE's contribution to understanding why clinical RL policies differ from observed practice. We appreciate your careful consideration.

Sincerely,

The Authors of TRACE

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.