- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This is a study on mechanistic interpretability in clinical decision support. It aims to explore whether clinical concepts (as defined by ICD - International Classification of Diseases) exist as locatable, causally used representations inside open-weight LLMs. Clinical notes are used aa dataset and expert-assigned ICD billing codes are used as as clinically validated proxies for automated learning of clinical concept centers. The study also conduct a blinded clinician validation and find the activation and usage of these concept centers predicts clinicians preferences.
Critical strengths
- The study aims to solve an impactful challenge.
- Involves eleven open models across Qwen and Gemma, including base, instruction-tuned, medical, dense and mixture-of-experts variants, which makes the findings less specific to one model
Critical weaknesses
-
Limited novelty, with overstated claims. The method derives a concept direction from labeled examples and tests it by injection, which follows known recipes such as concept activation vectors (TCAV) and causal concept effects (CaCE). The study is also close to concept-based models(e.g., Causal Concept Graph Models) which support concept-level interventions, but are not discussed. The abstract’s claim that mechanistic evaluation “has not been explored in clinical decision support” is contradicted by the paper’s own citations (Modi et al., 2026; Fraile Navarro et al., 2026). Furthermore, nothing in the approach seem to be clinical-specific; e.g., later in the text there is the statement that the method extends to any domain that maintains an established ontology. Therefore, my understanding is that the contribution is applicative, mostly evaluated on a single corpus (MIMIC-IV).
-
The causality experiments show that the directions can steer the output, not that the model relies on them. This conflicts with the claims that the concept centers “causally drive model behavior” (abstract) and are “causally used by the models.”
Questions for authors
- How does the method differ from concept activation vectors (TCAV) and causal concept effects (CaCE)? How does it relate to concept-based models?
- Тhe abstract states that mechanistic evaluation has not been explored in clinical decision support, yet §2 cites Modi et al. (2026) and Fraile Navarro et al. (2026). How does this work differ from them? What in the approach is specific to the clinical domain, beyond the availability of ICD labels?
- Do the results hold on a second corpus, e.g. notes from another institution or ICD-10-coded notes? 4.Section 2 states that the erasure machinery of Belrose et al. (2023) is followed on the removal side, but I was not able to locate removal results. Did you perform erase test? e.g., where chapter direction is erased and accuracy is measured?
- I was not able to understand Table 1. §4.3 describes several controls (e.g., permuted labels, wrong chapter, random direction, unembedding), but the table does not show them; it reports only a “Chapter-specific” column, in which the random-direction effect is subtracted rather than shown. Could you report, for every model at the main configuration, the results of each control and of negative strengths, and explain each column of the table?
5 comments
Author Response Part 2
Regarding the clinical contribution
We would like to emphasize that our formulation, realistic simulations, and expert validation are all specific to the clinical domain. The scope of the study is to investigate clinical concept centers in LLMs. We do not propose general methods for locating concepts, but clinical specific approaches that reformulate and scale interpretability methods in this domain with meaningful realistic uses: expert-assigned labels to scale concept discovery, simulations of clinical use under realistic priors, and validation with clinicians. We argue that for LLMs used as medical devices, these formulations and the insights they yield are important for extending the evaluation of models in the latent space beyond explaining specific failure modes.
Regarding the applicative nature
As explained earlier, prior work aimed to explain generally inert model representations behind specific failure modes. Our study instead leverages the nature of clinical data and real-world clinical behavior to scale interpretability beyond such representations, toward latent-space evaluation and steering of LLMs used in clinical settings.
Our contribution is a framework for scaling mechanistic interpretability to the clinical domain, with rigorous clinical grounding, cross-model evidence, and the combination of intervention, erasure, transfer and clinician validation. In the process, we introduce a scalable, expert-grounded adaptation of difference of means into the SAE feature basis, which automates locating concepts rather than only finding steering vectors.
Q3: Results on a second corpus, and erasure
We tested the concept centers, fitted only on MIMIC-IV discharge summaries, on MTSamples (a public corpus of 1,069 transcribed clinical reports from a different source and author population, labelled by medical specialty). Without refitting, the concept centers of Qwen3.5-27B separate the seven chapters with a mean one-vs-rest AUC of 0.856, compared to 0.868 on the original clinical note corpus. Thus it retains retains 97% of the in-corpus separability (permutation p < 1/2,000). Notably, this transfer holds even though specialty labels are coarser than the primary ICD codes the concept centers were fitted on, so the true separability is likely higher still. The concept centers also transfer to patient-question note excerpts (Appendix H) and separate orthopedic from hematology conversations in HealthBench (AUC 0.77). We will add these results to the appendix.
Erasure results
We thank the reviewer for their careful reading. We would like to note that the erasure experiments were conducted but unintentionally left out of the submitted version appendix due to a compilation error. We apologize for the oversight and report them here. We used the full-depth LEACE erasure (Belrose et al., 2023), as mentioned in the paper methodology. On Qwen3.5-9B-Base we fit a LEACE eraser at every layer on the selection notes, which removes all linearly decodable chapter information, and apply it at every layer and token position while the model answers the principal diagnosis question on held-out notes. As a control, we fit the same erasers on shuffled chapter labels, which remove a subspace of the same size which do not contain the concept.
| Condition | Full note | Diagnosis section removed |
|---|---|---|
| No erasure | 54.8% | 53.5% |
| LEACE, true chapter labels | 8.1% | 8.1% |
| LEACE, shuffled labels (control) | 45.3% | 41.4% |
Erasing the chapter concept reduces accuracy to near chance, at which point the model can no longer distinguish the chapters, while erasing a subspace of the same size without relevant concept only reduces performance by 10–12 points (this is expected, since there is some influence of these concept centers on adjacent concepts, as we have described in the paper). The results are consistent when the note's discharge-diagnosis section is also removed (which means that the model is not answering by matching diagnosis words to the options). These results show that linearly encoded chapter information is functionally necessary for this task and complement the steering results. We will add these results to the appendix.
W2: Causal use vs Causal Influence
Our claim is that the concept centers are used by the models, and that steering along them causally influences model behavior. As discussed, the erasure results we report above provide direct evidence for the former, showing that the features are causally used by the model for meaningful generation in clinical settings. Our clinician validation adds further empirical evidence: the strength with which a model activates the relevant concept centers while generating predicts clinician preference better than text-based instructions such as role-based priming. We believe this provides evidence that the concept centers are causally used by the models, and that steering along them causally influences model behavior.
Author Response Part 3
Q4: Questions regarding Table 1
Due to space constraints, the control results are reported across Table 1 and Appendix G. The random direction control is the Chapter-Specific column of table 1 (ie, ΔP minus the mean effect of random directions at the same layer and strength). We report the per-model random effects in Appendix G. The other controls are reported in lines 325 to 332 in the main text, with the details provided in Appendix G. We will make these pointers explicit in the caption and include their discussion and results in the main paper in the revised manuscript.
Regarding the use of AI Agents
For MIMIC-IV-related processing, we followed the referenced PhysioNet and MIMIC guidance (https://github.com/MIT-LCP/mimic-code/discussions/1593, https://physionet.org/news/post/gpt-responsible-use/), where local harnesses connected to Azure OpenAI endpoints were used for the note-reformatting step. All model experiments on MIMIC-IV were performed with locally hosted models. Cloud-based coding agents were not given access to MIMIC-IV data, model outputs derived from those data, or dataset artifacts. We will expand the AI-use statement with more specifics.
We hope these changes, along with the new results, analyses and clarifications above, address the reviewer’s concerns. We would be grateful if the reviewer would reconsider their score in light of them, and we are happy to provide any further clarification.
Author Response Part 1
W1 & Q1: Relation to TCAV, CaCE, and concept-based models
We thank the reviewer for these references. We would also like to clarify that our scope is to investigate the presence of clinical concept centers in LLMs, not to propose a new methodology for finding concept centers in LLMs. Our approach and novelty in experiments, ablations and real world simulations are all centred around designs which are faithful to the clinical setting, and we only propose that similar formulations might be useful to other domains with an established ontology as future work.
In addition, these lines of work references differ from ours both in their setting and in what they establish:
The scope of our study aims to answer a different question: whether clinical concepts already exist in, and influence, existing open-weight LLMs that were never trained with concept supervision. We study it post hoc on frozen LLMs with an existing learned representations that are not interpretable. The contribution is therefore not a new direction-extraction method. It is:
(i) the formulation of EHR notes with expert-assigned ICD codes as a labeled corpus for dictionary-based discovery,
(ii) the finding that such concept centers exist, are specific, persist through generation, and influence behavior across 11 open models, and
(iii) their use for latent-space evaluation, monitoring, and steering in clinical settings, with clinician validation.
Q2. Relation to Modi et al. and Fraile Navarro et al., and what is clinical-specific
Both works use SAEs to explain specific failures of clinical LLMs, but our work aims to address the open question of whether model latents contain clinical concepts as organized, specific representations that causally influence their clinical behavior.
Crucially, both works start from one known clinical error and trace it into the latent space, while we characterize clinical concepts across the full diagnostic taxonomy and establish that models use them, which is the foundation such targeted analyses rest on. Beyond the primary scope of explanation in prior work, the concept centers we find support latent-space evaluation, per-token monitoring, steering, and prediction of clinician preference.
We present a detailed breakdown of the scope of the studies below.
We will revise the abstract text to specifically state that prior mechanistic work in clinical decision support has focused on explanation of representations during specific failure modes instead of establishing the presence and use of clinical concepts, and make the distinction clearer in the related works.