- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper investigates how to create faithful natural language explanations for differences in action selection between two RL policies. The paper proposes TRACE, a framework for training an LLM that produces such explanations. For two RL policies using the same state encoder, TRACE learns a concept dictionary using top-k SAE on the encoder. The dictionary maps each feature in the SAE to some interpretation. Each feature is interpreted by comparing the training subjects across different activation values of that feature. The explainer LLM is initialized from Llama-3.1-8B-Instruct, SFT’d on teacher explanations, then PPO’d. For SFT, the teacher explanations are generated by GPT5.6-terra, using the patient state and selected concepts as input. For PPO, a frozen reader LLM scores whether the explanation helps recover the action. The paper compared TRACE against naive baselines and other existing policy-explanation methods, including methods with and without additional training. TRACE attains the highest F1 score for action prediction across all three tasks and both policies.
Critical strengths
The question that the paper tries to address is an interesting one. As more clinical ML systems are deployed, it is crucial for clinicians to understand the reasoning behind the recommended action from the RL policy, when it diverges from the population policy.
Critical weaknesses
-
In order to build the shared concept dictionary, TRACE requires the policies to use the same encoder. This in turn necessitates the training the encoder and policies from scratch. Also the framework only targets RL policies. Both points limit the general applicability of the framework.
-
The paper used general purpose LLMs for the explainer and the reader, as well as baseline in the evaluations. Given that we are in the clinical domain, it is important that we also test the effectiveness of the framework on specialized clinical LLMs (e.g. medgemma), and against baseline methods with these clinical LLMs.
Questions for authors
-
On L392-393, how is it determined whether an explainer cites a concept? On L410-411, how is “policy-selected cards” defined?
-
Across Table 1 and 2, it seems like the RL policy is noticeably more difficult to explain than the base policy. Do you have a hypothesis for why that might be the case?
-
To address the second applicability issue, instead of training the state encoder, have you tried using a pretrained medical sequence encoder (e.g. CLMBR-T)?
-
Right now the reader is about as capable as the explainer, and there is significant gap between the reader LLM and the judges used for evaluation. Have you tried varying the capability of the reader, relative to the explainer?
4 comments
Response to Reviewer TaVo (continued)
Q3 (pretrained encoder).
We thank the reviewer for the suggestion. However, CLMBR-T [2] reads coded Stanford timelines through its own vocabulary; mapping MIMIC's numeric vitals and labs into it is not supported by public tooling and would change the policies. We used a frozen pretrained ICU encoder that matches our data, BAT [3, 4] (pretrained on eICU and MIMIC-III), plus two controls, and ran TRACE's representation stages unchanged on RRT:
| paper | RL-only encoder | self-supervised | frozen BAT | |
|---|---|---|---|---|
| actions kept by SAE, RL/beh. (%) | 99.1/99.6 | 99.0/99.8 | 98.5/99.1 | 97.4/98.1 |
| named features replicating (%) | 98.1 | 97.7 | 96.1 | 96.1 |
| RL flip, 8 selected vs 8 random (%) | 39.5 vs 18.6 | 25.1 vs 14.8 | 15.0 vs 6.5 | 30.1 vs 12.8 |
The selected cards stay decision-relevant on every encoder, so TRACE needs a shared representation, not a particular training recipe. The BAT policy reads 24 h of history and is a different policy (63% action agreement), and we did not retrain the explainer on these variants.
Q4 (reader capability).
The paper's reader is Qwen3-8B: it scores every sentence of every rollout, so it must be small; gains are measured by a judge PPO never saw. Seven readers (Qwen3-1.7B to 32B, MedGemma-4B/27B) re-scored the frozen RRT explanations. The best (32B) recovers 0.538 / 0.533 of RL / behavior actions against 0.755 / 0.935 for Opus 5. Yet where the policies differ (119 states), the matching explanation beats a transplanted one on the RL target (0.571 vs 0.277 at 8B). Moreover, two independent judges, Opus 5 and GPT-6 Astra, exhibit substantial state-level disagreement (preference correlation 0.15) despite broadly consistent aggregate results. Such disagreement is not unexpected, as the evaluation judges serve as proxies for clinical experts, who may also differ in their assessments of individual cases. Nevertheless, this highlights the importance of accounting for judge specific biases, which we recognize as an important direction for future work.
W2: General-purpose vs. clinical LLM explainers on RRT (600 test states; same evaluation judge).
| Explainer | Method | RL Acc. / F1 | Behavior Acc. / F1 |
|---|---|---|---|
| None | No explanation | 0.453 / – | 0.488 / – |
| Llama-3.1-8B | Direct LLM | 0.452 / 0.293 | 0.495 / 0.373 |
| Llama-3.1-8B | Zero-shot CoT | 0.438 / 0.291 | 0.457 / 0.382 |
| MedGemma-4B | Direct LLM | 0.422 / 0.351 | 0.342 / 0.321 |
| MedGemma-4B | Zero-shot CoT | 0.412 / 0.350 | 0.370 / 0.350 |
| MedGemma-27B | Direct LLM | 0.400 / 0.345 | 0.377 / 0.358 |
| MedGemma-27B | Zero-shot CoT | 0.443 / 0.358 | 0.395 / 0.346 |
| MedGemma-27B | CoT (2,048 tokens) | 0.463 / 0.365 | 0.397 / 0.421 |
| Llama-3.1-8B | TRACE w/o PPO | 0.742 / 0.672 | 0.897 / 0.908 |
| Llama-3.1-8B | TRACE (8B reader) | 0.755 / 0.688 | 0.935 / 0.926 |
| MedGemma-4B | TRACE w/o PPO | 0.690 / 0.626 | 0.837 / 0.857 |
| MedGemma-4B | TRACE (27B reader) | 0.697 / 0.630 | 0.912 / 0.899 |
Specialized clinical LLMs do not consistently outperform general-purpose models in zero-shot settings. Increasing MedGemma-27B's generation limit to 2,048 tokens yields only modest improvements, with RL accuracy (0.463) remaining close to the no-explanation baseline (0.453). In contrast, TRACE substantially improves MedGemma-4B's performance, achieving 0.697/0.912 RL/behavior accuracy. PPO further improves its behavior accuracy by +0.075 (95% CI: [+0.041, +0.112]), while the RL gain is smaller (+0.007, 95% CI: [−0.035, +0.052]). These findings suggest that policy-specific training is more important than clinical specialization alone for recovering policy decisions.
References
[1] Sellergren, A., et al., 2025. MedGemma technical report. arXiv preprint arXiv:2507.05201.
[2] Wornow, M., Thapa, R., Steinberg, E., Fries, J. and Shah, N., 2023. EHRSHOT: An EHR benchmark for few-shot evaluation of foundation models. Advances in Neural Information Processing Systems, 36.
[3] DeVries, R., Christensen, C., Zepeda Mendoza, M.L. and Winther, O., 2025. Bi-axial transformers: Addressing the increasing complexity of EHR classification. arXiv preprint arXiv:2508.12418.
[4] Jagd, K.N., DeVries, R. and Winther, O., 2025. Towards self-supervised foundation models for critical care time series. arXiv preprint arXiv:2509.19885 (NeurIPS 2025 Workshop on Learning from Time Series for Health).
Q2 response is speculative. Keeping score
I'd like to thank the authors for the extensive responses to the points in my review. However, the response to Q2 remains speculative without empirical attribution or failure-mode analysis. In particular, showing lower action predictability from the state (e.g., via logistic regression) conflates policy complexity with post-hoc explainability, leaving open why generated explanations fail when the RL policy diverges from standard clinical practice. I'm going to keep my score.
We thank the reviewer for the follow-up. We agree that lower action predictability does not necessarily imply poorer explainability. To investigate the actual failure modes, we conducted an additional analysis on 600 RRT test states.
What causes lower RL explanation recovery? Without explanations, the judge predicts RL and behavior actions at similar rates (0.453 vs. 0.488). With TRACE explanations, recovery reaches 0.755 for RL and 0.935 for behavior, suggesting that state-level policy complexity alone cannot explain the gap.
Our analysis identifies two main sources of RL explanation failures:
- Ambiguous action values: When supported actions have similar Q-values, the explanation may not clearly distinguish the selected action from its closest alternative. RL recovery increases from 0.546 to 0.794 between low- and high-Q-margin states.
2. Behavior-support constraints: BCQ may exclude a higher-Q action because it falls outside the clinician-supported action set. Since explanations focus on clinical evidence rather than these constraints, the judge may favor the excluded action. As future work, we will investigate whether this limitation persists across different RL algorithms.
Together, these two failure modes account for 130 of 147 (88%) RL recovery errors. Importantly, 112 of these 147 errors occur when RL and behavior policies agree. Thus, the lower RL recovery is primarily associated with ambiguous action values and support constraints, rather than disagreement with clinical practice itself.
We hope this empirical analysis addresses the reviewer's concern.
Response to Reviewer TaVo
We thank the reviewer for the careful reading and constructive feedback. We have uploaded a revised PDF with changes highlighted in blue. Below, we address each point raised in the Weaknesses and Questions sections.
W1.
TRACE requires a shared encoder to place both policies in a common representation space, enabling the SAE to construct a unified concept dictionary for comparing their decisions. However, this does not require training the encoder or policies from scratch. The encoder can be pretrained and frozen, with policy heads trained independently. Moreover, TRACE is not restricted to RL policies: our behavior policy is trained through supervised learning using an adapter and linear head optimized with cross-entropy loss. Only the RL-specific concept ranking relies on Q-values and the support rule.
We have clarified these requirements in Section 2.2 and introduced three encoder variants (Q3) demonstrating that the shared representation need not be learned jointly (an RL-trained encoder with a behavior head fitted afterward), trained through RL (a self-supervised encoder learned without policy objectives), or trained from scratch (BAT, a frozen ICU encoder pretrained on eICU and MIMIC-III). We also acknowledge that TRACE does not directly support policies without an accessible, compatible representation, such as gradient-boosted trees, and have added this limitation to the paper.
W2 (Clinical LLMs).
We thank the reviewer for this suggestion. We conducted additional experiments using MedGemma-4B and MedGemma-27B [1] as explainers on the RRT test set, comparing them with Llama-3.1-8B under both zero-shot baselines and TRACE. The results report action recovery accuracy (RL / behavior):
At the standard token limit, neither MedGemma model surpasses the no-explanation judge baseline (0.453/0.488) in the zero-shot settings, suggesting that clinical knowledge alone is insufficient to recover policy-specific decisions. However, after training with the TRACE pipeline using a 27B reader, MedGemma-4B achieves 0.697/0.912, approaching Llama-3.1-8B (0.755/0.935), with gaps of only 0.058/0.023. These results demonstrate that TRACE can effectively extend to specialized clinical LLMs, with substantial improvements over their zero-shot baselines. Due to computational constraints, we did not conduct TRACE post-training for MedGemma-27B.
Q1.
We identify cited concepts with a rule-based text matcher that uses no LLM or other learned model. Each concept card specifies clinical variables and their directions (e.g., higher creatinine with lower urine output); the matcher finds these in the explanation through a predefined list of aliases and attributes each matching statement to the card that asserts it. A card counts as cited if its name is explicitly mentioned or a consecutive run of directional statements matches it. Against the card use the teacher recorded (19,096 RRT records) it has precision 0.936. Cp(s) is the set of active concepts with the highest score combining contribution to the representation, effect on the policy's action scores and effect on the margin between the selected action and the runner-up, with a redundancy penalty. Both definitions are now in the main text.
Q2 (Why RL is harder to explain).
We appreciate this insightful question. While we do not yet have a definitive explanation, we hypothesize that several factors may contribute to this gap.
First, the behavior policy closely reflects observed clinical practice, which may align better with the prior clinical knowledge of pretrained LLMs. Second, RL decisions appear more complex and less predictable from the state alone: on RRT, logistic regression recovers behavior actions with 0.901 accuracy, compared with 0.819 for RL actions, and the judge without an explanation shows a similar pattern across all three cohorts. Third, RL decisions depend on both Q-values and the support rule, with relatively small action margins (median Q-margin 0.14 versus Q-value SD 1.16).