acceptodds
Under review as a conference paper at ICLR 2027

Clinical Concept Centers In Large Language Models

Abstract

Large language models are increasingly used in clinical settings. However, research into the reliability and performance of these models has focused almost entirely on the language substrate, scoring what the model says, while the latent computations that produce these answers stay unexamined. At the same time, mechanistic interpretability has found that the latent space carries a higher fidelity of representation than the text: internal representations not only encode substantially more than the output verbalizes, but the stated reasoning also systematically omits features that causally drive the answer. An evaluation of model behavior in terms of mechanistic interpretability has not been explored in clinical decision support. We believe that it is a necessity, not only for improving model performance, but also to evaluate the safety and alignment of these models as medical devices. In this work, we extend behavioral evaluation into the latent space and ask whether clinical concepts exist as locatable, causally used representations inside open-weight LLMs. Reliable mechanistic interpretability in such settings uses dictionary learning over a labeled text corpus. We reformulate the EHR setting into exactly this approach, with clinical notes as the corpus and expert-assigned ICD billing codes as clinically validated proxies for automated learning of clinical concept centers. We find dedicated clinical concept centers in the latent space of all eleven open models we test. These concept centers are interpretable, firing only on their aligned clinical narratives, and meaningfully and causally drive model behavior in both constrained and open-ended settings. These concept centers are not just analytical representations, but circuits that can be utilized in clinical practice, and we explore their use from the perspective of both evaluation and performance. From the evaluation standpoint, models stay internally coherent and keep using the relevant concept centers even under adversarial role-based priming, while aligned priming improves downstream clinical performance. From a performance perspective, we simulate realistic deployment settings and find that steering models along these centers leads to meaningful downstream improvements. Finally, we conduct a blinded clinician validation and find the activation and usage of these concept centers predicts clinicians preferences.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, healthcare

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_68tCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is a study on mechanistic interpretability in clinical decision support. It aims to explore whether clinical concepts (as defined by ICD - International Classification of Diseases) exist as locatable, causally used representations inside open-weight LLMs. Clinical notes are used aa dataset and expert-assigned ICD billing codes are used as as clinically validated proxies for automated learning of clinical concept centers. The study also conduct a blinded clinician validation and find the activation and usage of these concept centers predicts clinicians preferences.

Critical strengths
  • The study aims to solve an impactful challenge.
  • Involves eleven open models across Qwen and Gemma, including base, instruction-tuned, medical, dense and mixture-of-experts variants, which makes the findings less specific to one model
Critical weaknesses
  1. Limited novelty, with overstated claims. The method derives a concept direction from labeled examples and tests it by injection, which follows known recipes such as concept activation vectors (TCAV) and causal concept effects (CaCE). The study is also close to concept-based models(e.g., Causal Concept Graph Models) which support concept-level interventions, but are not discussed. The abstract’s claim that mechanistic evaluation “has not been explored in clinical decision support” is contradicted by the paper’s own citations (Modi et al., 2026; Fraile Navarro et al., 2026). Furthermore, nothing in the approach seem to be clinical-specific; e.g., later in the text there is the statement that the method extends to any domain that maintains an established ontology. Therefore, my understanding is that the contribution is applicative, mostly evaluated on a single corpus (MIMIC-IV).

  2. The causality experiments show that the directions can steer the output, not that the model relies on them. This conflicts with the claims that the concept centers “causally drive model behavior” (abstract) and are “causally used by the models.”

Questions for authors
  1. How does the method differ from concept activation vectors (TCAV) and causal concept effects (CaCE)? How does it relate to concept-based models?
  2. Тhe abstract states that mechanistic evaluation has not been explored in clinical decision support, yet §2 cites Modi et al. (2026) and Fraile Navarro et al. (2026). How does this work differ from them? What in the approach is specific to the clinical domain, beyond the availability of ICD labels?
  3. Do the results hold on a second corpus, e.g. notes from another institution or ICD-10-coded notes? 4.Section 2 states that the erasure machinery of Belrose et al. (2023) is followed on the removal side, but I was not able to locate removal results. Did you perform erase test? e.g., where chapter direction is erased and accuracy is measured?
  4. I was not able to understand Table 1. §4.3 describes several controls (e.g., permuted labels, wrong chapter, random direction, unembedding), but the table does not show them; it reports only a “Chapter-specific” column, in which the random-direction effect is subtracted rather than shown. Could you report, for every model at the main configuration, the results of each control and of negative strengths, and explain each column of the table?
5 comments
AuthorsOpenReview ↗

Author Response Part 1

W1 & Q1: Relation to TCAV, CaCE, and concept-based models

We thank the reviewer for these references. We would also like to clarify that our scope is to investigate the presence of clinical concept centers in LLMs, not to propose a new methodology for finding concept centers in LLMs. Our approach and novelty in experiments, ablations and real world simulations are all centred around designs which are faithful to the clinical setting, and we only propose that similar formulations might be useful to other domains with an established ontology as future work.

In addition, these lines of work references differ from ours both in their setting and in what they establish:

  • TCAV learns a concept activation vector from user-supplied example sets in a trained image classifier and measures the gradient sensitivity of a class score along it. It does not add the direction to the model's activations and observe how its outputs change. We will cite it in prior work since it attempts to estimate a concept direction from labeled examples
  • CaCE defines a concept's causal effect as an interventional difference in a classifier's output, estimated with counterfactual images from a controlled generator or a conditional VAE. It intervenes on only on inputs, not internal activations, and is evaluated on image classifiers (not the language domain).
  • Causal Concept Graph Models are meant to be interpretable by design: they are trained with concept labels and learn a causal graph over those concepts, and their interventions act on the model's own predicted concepts. They do not investigate methods for interpreting existing learned representations, like those of LLMs.

The scope of our study aims to answer a different question: whether clinical concepts already exist in, and influence, existing open-weight LLMs that were never trained with concept supervision. We study it post hoc on frozen LLMs with an existing learned representations that are not interpretable. The contribution is therefore not a new direction-extraction method. It is:

(i) the formulation of EHR notes with expert-assigned ICD codes as a labeled corpus for dictionary-based discovery,

(ii) the finding that such concept centers exist, are specific, persist through generation, and influence behavior across 11 open models, and

(iii) their use for latent-space evaluation, monitoring, and steering in clinical settings, with clinician validation.

Q2. Relation to Modi et al. and Fraile Navarro et al., and what is clinical-specific

Both works use SAEs to explain specific failures of clinical LLMs, but our work aims to address the open question of whether model latents contain clinical concepts as organized, specific representations that causally influence their clinical behavior.

Crucially, both works start from one known clinical error and trace it into the latent space, while we characterize clinical concepts across the full diagnostic taxonomy and establish that models use them, which is the foundation such targeted analyses rest on. Beyond the primary scope of explanation in prior work, the concept centers we find support latent-space evaluation, per-token monitoring, steering, and prediction of clinician preference.

We present a detailed breakdown of the scope of the studies below.

Fraile Navarro et al.Modi et al.Ours
Analysis scopefinding medical-vs-non-medical features under triagefind word-sense latents of 12 terms during unstable oncology staging17 organ-system clinical concepts
Nature of representations foundDecodable, but with no measurable effect on decisions (interventions change 0–2 of 60 predictions)Needed for next-token prediction, but no effect on clinical decisions testedDecodable and evidence of causal use and influence during clinical decision making
Depth of latent analysis4 layers in 2 modelsOne SAE on 1 layerEvery layer of 11 models with decodability and causal influence
Grounding of conceptscontrastive promptsauthor inspection of top tokensexpert-assigned ICD codes on 8,617 notes
Effect on clinical behaviornear zero impactnext-token loss only, with no decision level studyeffect on clinical decisions across 11 models, with four controls and held-out selection
Tracing During generationprompt tokens only64-token note windowstracked over up to 2,000 generated tokens
Clinician involvement16 cases to calibrate output labelsphysicians authored the casesusing concept centers to predict clinician preferences + scalable expert label grounding

We will revise the abstract text to specifically state that prior mechanistic work in clinical decision support has focused on explanation of representations during specific failure modes instead of establishing the presence and use of clinical concepts, and make the distinction clearer in the related works.

AuthorsOpenReview ↗

Author Response Part 2

Regarding the clinical contribution

We would like to emphasize that our formulation, realistic simulations, and expert validation are all specific to the clinical domain. The scope of the study is to investigate clinical concept centers in LLMs. We do not propose general methods for locating concepts, but clinical specific approaches that reformulate and scale interpretability methods in this domain with meaningful realistic uses: expert-assigned labels to scale concept discovery, simulations of clinical use under realistic priors, and validation with clinicians. We argue that for LLMs used as medical devices, these formulations and the insights they yield are important for extending the evaluation of models in the latent space beyond explaining specific failure modes.

Regarding the applicative nature

As explained earlier, prior work aimed to explain generally inert model representations behind specific failure modes. Our study instead leverages the nature of clinical data and real-world clinical behavior to scale interpretability beyond such representations, toward latent-space evaluation and steering of LLMs used in clinical settings.

Our contribution is a framework for scaling mechanistic interpretability to the clinical domain, with rigorous clinical grounding, cross-model evidence, and the combination of intervention, erasure, transfer and clinician validation. In the process, we introduce a scalable, expert-grounded adaptation of difference of means into the SAE feature basis, which automates locating concepts rather than only finding steering vectors.

Q3: Results on a second corpus, and erasure

We tested the concept centers, fitted only on MIMIC-IV discharge summaries, on MTSamples (a public corpus of 1,069 transcribed clinical reports from a different source and author population, labelled by medical specialty). Without refitting, the concept centers of Qwen3.5-27B separate the seven chapters with a mean one-vs-rest AUC of 0.856, compared to 0.868 on the original clinical note corpus. Thus it retains retains 97% of the in-corpus separability (permutation p < 1/2,000). Notably, this transfer holds even though specialty labels are coarser than the primary ICD codes the concept centers were fitted on, so the true separability is likely higher still. The concept centers also transfer to patient-question note excerpts (Appendix H) and separate orthopedic from hematology conversations in HealthBench (AUC 0.77). We will add these results to the appendix.

Erasure results

We thank the reviewer for their careful reading. We would like to note that the erasure experiments were conducted but unintentionally left out of the submitted version appendix due to a compilation error. We apologize for the oversight and report them here. We used the full-depth LEACE erasure (Belrose et al., 2023), as mentioned in the paper methodology. On Qwen3.5-9B-Base we fit a LEACE eraser at every layer on the selection notes, which removes all linearly decodable chapter information, and apply it at every layer and token position while the model answers the principal diagnosis question on held-out notes. As a control, we fit the same erasers on shuffled chapter labels, which remove a subspace of the same size which do not contain the concept.

ConditionFull noteDiagnosis section removed
No erasure54.8%53.5%
LEACE, true chapter labels8.1%8.1%
LEACE, shuffled labels (control)45.3%41.4%

Erasing the chapter concept reduces accuracy to near chance, at which point the model can no longer distinguish the chapters, while erasing a subspace of the same size without relevant concept only reduces performance by 10–12 points (this is expected, since there is some influence of these concept centers on adjacent concepts, as we have described in the paper). The results are consistent when the note's discharge-diagnosis section is also removed (which means that the model is not answering by matching diagnosis words to the options). These results show that linearly encoded chapter information is functionally necessary for this task and complement the steering results. We will add these results to the appendix.

W2: Causal use vs Causal Influence

Our claim is that the concept centers are used by the models, and that steering along them causally influences model behavior. As discussed, the erasure results we report above provide direct evidence for the former, showing that the features are causally used by the model for meaningful generation in clinical settings. Our clinician validation adds further empirical evidence: the strength with which a model activates the relevant concept centers while generating predicts clinician preference better than text-based instructions such as role-based priming. We believe this provides evidence that the concept centers are causally used by the models, and that steering along them causally influences model behavior.

AuthorsOpenReview ↗

Author Response Part 3

Q4: Questions regarding Table 1

Due to space constraints, the control results are reported across Table 1 and Appendix G. The random direction control is the Chapter-Specific column of table 1 (ie, ΔP minus the mean effect of random directions at the same layer and strength). We report the per-model random effects in Appendix G. The other controls are reported in lines 325 to 332 in the main text, with the details provided in Appendix G. We will make these pointers explicit in the caption and include their discussion and results in the main paper in the revised manuscript.

Regarding the use of AI Agents

For MIMIC-IV-related processing, we followed the referenced PhysioNet and MIMIC guidance (https://github.com/MIT-LCP/mimic-code/discussions/1593, https://physionet.org/news/post/gpt-responsible-use/), where local harnesses connected to Azure OpenAI endpoints were used for the note-reformatting step. All model experiments on MIMIC-IV were performed with locally hosted models. Cloud-based coding agents were not given access to MIMIC-IV data, model outputs derived from those data, or dataset artifacts. We will expand the AI-use statement with more specifics.

We hope these changes, along with the new results, analyses and clarifications above, address the reviewer’s concerns. We would be grateful if the reviewer would reconsider their score in light of them, and we are happy to provide any further clarification.

Reviewer_gVYpOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that clinical LLMs are often judged only by their direct human-readable output. However, their internal (latent) representations arguably contain more useful information. The authors show that these patterns of features/representations (called concept centers) exist in common LLMs across all layers in those models with strong separability, and they also show that factoring those representations into the decision-making process of the LLM increases its probability of choosing the right answer. During generation of responses, they demonstrate that the representations are more or less “constantly active” for each word, although these levels seem to vary quite a lot for the Qwen models. Separately, the authors evaluate the impact of “role-prompting” (i.e. telling the model it is a specialist for a certain task), whether steering the model to use these representations helps in writing answers instead of choosing them via multiple choice, and whether clinicians prefer the higher-scored answers.

Critical strengths
  1. The paper’s results use a clean split of halves for selecting features and evaluating on held-out data. Further, they use a patient-level split, which should place all the information for one patient in one half, so no leakage can occur. This leads me to believe that the reported numbers are valid and generalize within their experiments/dataset.

  2. The random-direction control in table 1 seems to show convincing evidence that their identified chapter directions actually contain meaningful information.

Critical weaknesses
  1. One of the main points the authors are trying to make is that the internal representations of the model are critical for improving model performance (or even more important than the raw text output). However, it doesn’t seem like there is any evaluation/comparison of the text output itself. For example, in figure 1, would that achieve a similar AUC? Similarly in the results about steering the model, it’s not clear whether steering via the chapter direction is necessary instead of just adding that meta information (i.e. department) into the prompt. Overall, many of the results are difficult to contextualize due to these choices.

  2. The main text seem to focus on less important results. In figure 2, there doesn’t seem to be any explanation of why certain model curves vary so much across word generations, and it’s not clear what these curves would look like if they were generating off-target words (i.e. some kind of control curve that should ideally differ significantly from the current ones). In contrast, section 6 presents the more interesting (and realistic) scenario of generating clinical answers instead of choosing them via multiple choice, but the main text section here feels much less developed. The way correctness is measured is not clear, and even in the corresponding appendix section, the gains look quite small (“+1.1 to +2.3 additional correct diagnoses per hundred cases”) and no confidence intervals are reported. The paper would benefit from looking into this scenario with more detail.

Questions for authors
  1. A lot of the text feels very difficult to read, and I found there to be quite a bit of jargon throughout. In particular, the introduction feels very text-heavy, with little separation or paragraph titles. The paper would benefit a lot from some kind of overall summary figure showing what these concept centers look like and what part of a conventional model they represent.

  2. The text under table 1 mentions four different controls, but I only saw the random one mentioned in the table as the “chapter-specific” gain. Near equation 1, not all variables are defined, like W_dec and b_dec.

7 comments
AuthorsOpenReview ↗

Author Response Part 1

We thank the reviewer for their review and raising important concerns. We address them below.

W1 Text baselines and comparison with the language outputs

We added direct text-output baselines to contextualize the latent-space results. We present some analysis and findings on our strongest chat model, Qwen3.5-27B, to better contextualize the necessity and utility the concept centers.

Separability (Figure 1). The model's own text answer distribution gives an AUC of 0.877 on the same notes, as compared to 0.904 for the concept centers we find. The interpretable latent features which activate when the model simply reads the note therefore contain as much information as the model’s actual generated answer to the classification question.

In addition, we find phenomenon where models which produce poor textual outputs still do contain the information required for the task in their latent space. As an example, II-Medical-8B answers correctly on only 11.3% of notes (output AUC 0.588), but the concept centers still separate the chapters at 0.902. This provides a signal on models capabilities on the task which cannot be judged from its text output. As we demonstrate in our results, steering these concept centers in the model more than doubles the probability of the correct answer, a chapter-specific gain several times the random-direction floor (the details are available in Appendix J).

Clinician preference. A text-level score of how much specialty vocabulary a response contains poorly predicts clinician preference (40.2% agreement, p = 0.010), while concept-center activation predicts it positively. This points to another utility of the concept centers, where we can analyze preferred responses based on how much they engage the specialty concept internally without specifically verbalizing it in the vocabulary of the model. The concept-center activation therefore contains information associated with clinician preference beyond these measured topic and length baselines.

Department signal in the prompt vs steering latent concepts. We compare the signal from adding metadata in the prompt to steering the model using the concept centers. To capture realistic settings, we also simulated a department prior that is wrong on 20% of notes and compared steering against simply writing the department into the prompt. We present the results on the free-form diagnosis task below (Qwen3-8B-Base, 1,624 notes):

ConditionChange in diagnosis accuracy
Department in the prompt+4.1 [+2.9, +5.4]
Steering along the concept center+2.4 [+1.3, +3.6]
Both+8.2 [+6.7, +9.8]
Department in the prompt, 20% of departments wrong−1.6 [−3.3, +0.1]
Steering, 20% of departments wrong+2.1 [+0.9, +3.3]

With an imperfect prior, writing the department into the prompt reduces performance below the unsteered model, while steering keeps nearly all of its gain. Steering also improve performance further when combined with the prompt input. The concept centers therefore provide a way to steer models that is both complementary to prompting and robust to the errors in realistic deployment metadata.

These results support our claim that latent-space evaluation complements (rather than replaces) the output-level evaluation and can reveal task-relevant information that is not reliably expressed in the generated answer.

Causal use and the relevance of the concept centers

To further contextualize the relevance of these concept centers in the models text generation, we also present our erasure experiments (which were unintentionally omitted from the appendix due to compilation issues). In these experiments, we removed the chapter concept from Qwen3.5-9B-Base with LEACE (Belrose et al., 2023). We fit one eraser per layer and applied it at every layer and token position. Any linear representation of the diagnosis, including the concept centers, then loses its this information. As a control we used erasers fitted on permuted chapter labels, which remove the same number of dimensions but no relevant chapter information. The model is then tasked with the same primary diagnosis classification.

ConditionFull noteDiagnosis section removed
No erasure54.8%53.5%
Permuted-label erasure45.3% (−9.6)41.4% (−12.1)
Chapter erasure8.1% (−46.7)8.1% (−45.4)

As we can see in these results, erasing the chapter concept in the latent space drops models generated answer accuracy nearly to random chance. This remains true even when we remove diagnosis related sections from the note. Thus, the models text output for this task is dependent on the use of the information represented in these concept centers. This provides direct evidence beyond steering that the concept centers are not just features the latent space happened to encode but representations which influence the text generation of the model.

AuthorsOpenReview ↗

Author Response Part 2

Qualitative analysis on the impact of steering on text outputs

Steering the model based does change the clinical finding meaningfully. We share some qualitative examples for different models below.

Principal diagnosis chapterUnsteeredSteered along the concept centerComments
Infectious diseases (Qwen3-8B-Base)“Esophageal cancer in remission with complications of cirrhosis, ascites, and pleural effusions, likely due to underlying liver disease and hepatitis C.”“Severe sepsis and septic shock, likely due to a combination of pneumonia, liver disease, and possibly a gastrointestinal infection, leading to hypotension…”The unsteered model summarizes the patients oncologic history, while the steered model identifies the acute infection which led to the admission
Blood disorders (Qwen3-8B-Base)“Chronic anemia, Hepatitis C cirrhosis, Esophageal varices, Angioectasias, Polyps in stomach antrum, Diabetes mellitus, Chronic kidney disease.”“Chronic anemia due to chronic blood loss, cirrhosis with varices, and diabetes mellitus.”The unsteered model provides the entire problem list while the steered model focuses on the principal finding and its cause.
Neoplasms (Gemma-3-4B-it)“Moderate pulmonary artery hypertension with features suggestive of chronic pulmonary emboli.”“The patient has myelofibrotic hypercellular marrow with abnormal megakaryocytes consistent with a myeloproliferative neoplasm.”The model goes from a secondary cardiopulmonary finding to the myeloproliferative neoplasm underlying the admission.

We will add this qualitative analysis in the revised paper to better highlight the utility beyond quantitative gains.

Generalization to clinical text from other sources.

For the concept centers to be useful in deployment, they must work on clinical text beyond the distribution they were discovered on. To further demonstrate their utility, we applied them without refitting to MTSamples (a public corpus of 1,069 transcribed clinical reports from a different source and author population, which are labelled by medical specialty). The concept centers of Qwen3.5-27B separate the diagnostic chapters with a mean AUC of 0.856, compared to 0.868 on the original clinical notes. They thus retain 97% of their separability (permutation p < 1/2,000). This provides evidence of cross-source and cross-register transfer without refitting, despite differences in documentation style and label granularity, which is what monitoring and steering in practical deployment require.

We hope these experiments, analysis and results can meaningfully contextualize the relevance, necessity, generalization and utility of the concept centers better. We remain open to further suggestions which can help improve the presentation of our results.

W2. Emphasis of the main text and the generation setting

In the main text, we wanted to present the key process of finding the concept centers, validating their decodability and causal influence and then presenting their use from both evaluation and performance perspective. In this context, we found emphasizing the foundational setting of establishing their existence and causal use more important and meaningful, given the space constraints. This allowed for the subsequent sections on evaluation and performance based utility of these concept centers to be better founded and convincing. Given that the scope of the study is to establish the presence and causal influence of the clinical concept centers, we found this to be a reasonable assumption.

However, we appreciate the reviewers interest in section 6 and agree that the performance steering using these concept centers is an important and practically meaningful result. We will use most of the additional space for camera ready to focus on the results we presented earlier to establish better necessity and utility of the concept centers. This includes a figure 1 (which we will add in the appendix of the revised manuscript for reference) the additional discussion on the realistic simulations we presented in section 6, along with the prompt based analysis and generalization beyond the dataset where we find these concept centers to other clinical benchmarks like MTSamples and Healthbench.

AuthorsOpenReview ↗

Author Response Part 3

We now address the specific questions and concerns in this weakness below:

Correctness measurement and confidence intervals. With the injection held active at every generated token, the model writes its own diagnostic impression. The impression is then appended to the note, and the primary diagnosis is asked, unsteered and without any department hint. Every condition is therefore scored with the same prompt and differs only in the text the model generated under steering. The paired 95% confidence intervals for the reported gains are [+0.1, +2.2] (Qwen3-8B-Base), [+0.9, +3.7] (Qwen3.5-9B-Base) and [−0.1, +2.5] (Gemma-3-4B-it) correct diagnoses per hundred cases. We will add these and the other technical details to the revised manuscript.

Variation across generations in Figure 2. This is discussed in Appendix I due to space constraints, and we summarize it here.

  • Length. Generations run to a natural ending, so fewer passes contribute at later token positions. II-Medical-8B stops at a median of 359 tokens, which is why its curve ends near token 500.
  • Content. The signal is spikier on some models. On II-Medical-8B the projection is near zero over the opening tokens and activates on chapter-aligned diagnostic tokens later in the window, so its curve follows what the model is writing.
  • Depth. The underlying projection grows with depth, so each model is read at its most responsive layer with near-complete coverage.

Beyond the appendix, most of the variance lies token to token within a generation as compared to between generations. Dominance is stronger where the text discusses the chapter, yet it stays positive in 69% of windows that never mention it (as discussed in W1).

Off-target control. We computed the control curve on all eleven models, at the same layers and over the same generations, by tracking a fixed, randomly drawn wrong chapter with the same dominance measure. The off-target curve stays near zero on every model (effect size −0.06 to +0.04), compared to +0.04 to +0.52 for the true chapter. The difference is significant on all eleven models (paired Wilcoxon over generations, p ≤ 0.009).

Q2: Controls in Table 1, and the variables near Equation 1

Due to space constraints, the control results are presented across Table 1 and Appendix G. The random-direction control is the Chapter-Specific column of Table 1 (i.e., ΔP minus the mean effect of random directions at the same layer and strength), and the per-model random effects are reported in Appendix G. The other controls are reported in lines 325 to 332 of the main text, with the details provided in Appendix G.

W_dec and b_dec are the SAE’s decoder matrix and decoder bias, which are introduced in the decoding step: h_hat = W_dec z + b_dec , just before Equation 1. Equation 1 uses only W_dec, because the bias cancels in a difference of decoded means. We will mention this explicitly when we introduce the variables.

Q1: Regarding writing

The paper uses terminology from both mechanistic interpretability and clinical NLP, which might make it demanding to follow given the amount of text. We wrote to what we expected the reader’s mental model to be, and we agree that some restructuring would make it more accessible.

We propose the following updates to make the text easier to follow:

  1. Overview figure: A summary figure to represent our key process (clinical data reformulation, foundational approaches to establish the existence of the clinical concept centers with exemplars on what they look like, followed by their utility from an evaluation and performance perspective). We will add this figure in the appendix for the revised manuscript, and move it to the main paper with the additional space provided in the camera ready.
  2. Paragraph titles and splitting some longer sentences
  3. Clinical terminology definitions on first use, and a glossary in the appendix

We hope these changes, along with the new results, analyses and clarifications above, address the reviewer’s concerns. We would be grateful if the reviewer would reconsider their score in light of them, and we are happy to provide any further clarification.

Reviewer_mqZfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper utilizes ICD-coded clinical note data to identify SAE-based concept directions in 11 LLMs. The paper then assesses whether these directions can differentiate between different types of diseases and how they affect the output of the models. Additionally, the paper evaluates category-guided steering and tests whether concept activation can predict clinician preference between two different role-prompted responses.

Critical strengths
  1. Utilizing existing clinical coding labels is a practical approach to studying disease-related representations. The use of multiple model families adds value to the findings.
  2. The intervention controls strengthen the analysis beyond probing alone. The finding that decodability and intervention effectiveness occur at different layers is particularly interesting.
Critical weaknesses
  • My main concern is the gap between the reported results and the claimed clinical utility. Steering improves chapter-level prediction when the true chapter is supplied, but this does not establish better clinical decisions using realistically available information. The clinician study provides useful evidence linking activation to preference, but compares role-prompted responses rather than steered and unsteered responses. It therefore does not directly validate the clinical benefits of steering.
  • The paper provides evidence that the SAE dictionaries capture broad clinical categories, including a fresh-dictionary check and comparisons showing stronger steering than raw directions. However, it remains unclear whether these category-level measurements are informative about clinical correctness beyond topic alignment. Their added value for response-quality assessment is not compared with raw projections or probe-based scores.
Questions for authors
  1. Is there evidence that the steering gains hold when directions are selected using independently available clinical metadata or predicted categories?
  2. Do SAE-based scores predict clinician preferences better than raw projections or probe-based scores?
  3. Appendix C appears to mismatch ICD-9 category names and ranges. For example, 390–459 is labeled Sense Organs rather than Circulatory System. Is this confined to the manuscript, or did it affect the experimental labels, prompts, or specialty assignments?
4 comments
AuthorsOpenReview ↗

Author Response Part 1

We thank the reviewer and address their concerns below.

Weakness 1 and Question 1: Clinical Utility

We would like to clarify that the scope of the study is not to establish steering for performance gains, but rather the presence and usage of clinical concept centers in LLMs, where erasing the clinical concept at every layer reduces the model's diagnosis accuracy to near chance (54.8% to 8.1%) while a same-size control erasure costs only 10–12 points. The empirical validation of the presence of such concept centers itself unlocks utility, which extends well beyond steering, from latent-space evaluation to monitoring in practical settings, as we demonstrate in different sections of the paper.

In that context, we note that our experiments demonstrate clinical utility of these latent concept centers (for both monitoring and performance steering) using realistically available information such as real clinical notes and information received by a deployed clinical model.

For example, LLMs in the real world are often provided clinical notes and asked to make clinical decisions. This makes it important to evaluate whether the model aligns with the existing steering methods such as role based priming, from a practical utility perspective. As we demonstrate in Section 5 of the paper, the model presents internal coherence where it relies on its latent concept centers more strongly than the provided role. This not only shows that the prompt-level steering does not reliably translate into response quality, but emphasizes on a general direction to better represent clinical concepts in the model latents, especially since these concepts persist through long form text generation of the models (as shown in section 4.4). We would argue that this mode of evaluation in itself has utility for real world usage of models as medical devices.

To focus on the steering based utility of these concept centers, we do go beyond chapter prediction. As described in Section 6, we evaluate steering in a realistic long-form setting where the model reads a full clinical note, and writes the principal diagnosis. We simulate the department prior a deployed model would have by steering along the concept center of the relevant department. This is a realistic setting which accounts for a large majority of LLM based tasks in clinical practice (https://pmc.ncbi.nlm.nih.gov/articles/PMC12716761/). Steering the model based in such a setting changes the clinical finding meaningfully. We share some qualitative examples for different models below.

Principal diagnosis chapterUnsteeredSteered along the concept centerComments
Infectious diseases (Qwen3-8B-Base)“Esophageal cancer in remission with complications of cirrhosis, ascites, and pleural effusions, likely due to underlying liver disease and hepatitis C.”“Severe sepsis and septic shock, likely due to a combination of pneumonia, liver disease, and possibly a gastrointestinal infection, leading to hypotension…”The unsteered model summarizes the patients oncologic history, while the steered model identifies the acute infection which led to the admission
Blood disorders (Qwen3-8B-Base)“Chronic anemia, Hepatitis C cirrhosis, Esophageal varices, Angioectasias, Polyps in stomach antrum, Diabetes mellitus, Chronic kidney disease.”“Chronic anemia due to chronic blood loss, cirrhosis with varices, and diabetes mellitus.”The unsteered model provides the entire problem list while the steered model focuses on the principal finding and its cause.
Neoplasms (Gemma-3-4B-it)“Moderate pulmonary artery hypertension with features suggestive of chronic pulmonary emboli.”“The patient has myelofibrotic hypercellular marrow with abnormal megakaryocytes consistent with a myeloproliferative neoplasm.”The model goes from a secondary cardiopulmonary finding to the myeloproliferative neoplasm underlying the admission.

We will add this qualitative analysis in the revised paper to better highlight the utility beyond predictive gains.

AuthorsOpenReview ↗

Author Response Part 2

Steering with imperfect, independently available information

To test whether the gains remain in realistic settings when the prior is not the true chapter, we simulated a department prior that is wrong on 20% of notes and compared steering against simply writing the department into the prompt, on the free-form diagnosis task (Qwen3-8B-Base, 1,624 notes):

ConditionChange in diagnosis accuracy
Department in the prompt+4.1 [+2.9, +5.4]
Steering along the concept center+2.4 [+1.3, +3.6]
Both+8.2 [+6.7, +9.8]
Department in the prompt, 20% of departments wrong−1.6 [−3.3, +0.1]
Steering, 20% of departments wrong+2.1 [+0.9, +3.3]

With an imperfect prior, writing the department into the prompt reduces performance below the unsteered model, while steering keeps nearly all of its gain. Steering also improves performance further when combined with the prompt input. The concept centers therefore provide a way to steer models that is both complementary to prompting and robust to the errors in realistic deployment metadata.

Generalization to clinical text from other sources.

For the concept centers to be useful in deployment, they must work on clinical text beyond the notes they were discovered on. To further demonstrate their utility, we applied them without refitting to MTSamples (a public corpus of 1,069 transcribed clinical reports from a different source and author population, labelled by medical specialty). The concept centers of Qwen3.5-27B separate the diagnostic chapters with a mean AUC of 0.856, compared to 0.868 on the original discharge summaries. Thus it retains 97% of the separability (permutation p < 1/2,000). This provides evidence of cross-source and cross-register transfer without refitting, despite differences in documentation style and label granularity, which is what monitoring and steering in practical deployment require.

Regarding the Clinician Study

Given that the key scope of the study is to establish the presence and use of clinical concept centers in LLM, our goal with the clinician study was to test the models use of these concept centers in relation to response quality, not whether steering helps. The goal is not to compare the responses of the models under role based priming, but instead to induce some variation in the model responses so that we can study the impact of the underlying model latents under this variation. We found this to be a more meaningful study to establish the core claims of the paper, since the presence and use of these concept centers on clinical outputs are the foundational findings which drive the utility we demonstrate across model evaluation and performance steering. We agree that a blinded clinician comparison of steered versus unsteered generations would directly test the clinical effect of steering. The present clinician study addresses a different question: whether naturally varying concept-center activation predicts clinician preference independently of the prompting condition. We will clarify this distinction and identify direct clinician validation of steering as future work.

AuthorsOpenReview ↗

Author Response Part 3

Weakness 2 and Question 2

We thank the reviewer for this question and evaluated it directly on the clinician-rated responses. For every response we computed several alternative scores from the same generations. We then applied the same zero-parameter rule used in §7 to each score: the response with the higher mean target-concept dominance is predicted to be preferred. All scores are evaluated on the identical 194 non-tie clinician judgments and the identical gap strata reported in the paper. The comparison covers four instruments:

  • the SAE-derived concept center, at the best representation and steering layers
  • the raw difference-of-means direction at the best representation and steering layers
  • a multinomial logistic probe fitted on the clinical notes and applied to every token of each response
  • two text-only baselines: musculoskeletal vocabulary density and response length.
Score used to predict clinician preferenceAll pairsTop half by activation gapTop quarter by activation gap
SAE concept center, representation layer55.7% (p = .13)66.0% (p = .002)76.0% (p < .001)
SAE concept center, steering layer62.4% (p = .001)71.1% (p < .001)76.0% (p < .001)
Raw difference of means, representation layer58.2% (p = .026)63.9% (p = .008)78.0% (p < .001)
Raw difference of means, steering layer52.6% (p = .52)58.8% (p = .10)58.0% (p = .32)
Logistic probe (best classification layer)46.4% (p = .35)43.3% (p = .22)36.0% (p = .065)
Text only: musculoskeletal vocabulary density40.2% (p = .010)37.0% (p = .016)38.0% (p = .12)
Text only: response length (longer preferred)46.9% (p = .43)36.1% (p = .008)30.0% (p = .007)

We note the key results below:

  • The concept-center signal goes beyond topic alignment. The topic-level scores do not predict clinician preference. This shows that if the concept-center measurement only tracked how much a response is topically or categorically contains the relevant specialty, it would not have predicted preference well. From the results, we see that measuring musculoskeletal topical content does not predict which response clinicians prefer. Responses with more musculoskeletal vocabulary are in fact significantly less preferred (40.2%). The same applies for the logistic probe (46.4%) as well. This is true even though the probe correctly identifies the responses as musculoskeletal: it ranks that chapter first for 42–53% of responses (vs random chance of 5.9%). The probe is also the most accurate chapter classifier of all the instruments we study (0.93 macro AUC). Thus, decoding the clinical topic is therefore not what predicts quality or clinician preference. This entails that the concept-center activation captures goes beyond the strongest topic classifier. The same holds during generation: the concept center remains active in 69% of 50-token windows whose text never mentions the chapter (Qwen3.5-27B).
  • The signal is not a topic or length effect. We regressed the clinician's choice jointly on the activation gap, the musculoskeletal-vocabulary gap, and the length gap. Activation remains significant and positive (z = +3.25, p = 0.001), vocabulary is significant and negative (z = −3.40), and length is not significant. Notably, activation and vocabulary are uncorrelated (r = 0.00). Agreement is the same whether the higher-activation response is the longer one (54.5%) or the shorter one (56.1%).
  • Comparison with raw projections. At the best representation layer, the SAE-derived and raw difference-of-means projections predict preference comparably. However, at the steering layer, only the SAE-derived score remains predictive (62.4% against 52.6% over all pairs). The SAE-derived signal is therefore more stable across depth.

We also note that beyond better encoding of clinical correctness, SAEs also provide an interpretability benefit. The features decompose into specific directions that can labelled and monitored token by token during generation. This is not possible with the raw direction or a probes.

Additionally, as the qualitative examples and clinical simulation shows, the steering and use of these concept centers helps in guiding models to more meaningful outputs during clinical decision support generation.

We will add this discussion in the revised manuscript.

Q3. Mismatched ICD-9 names and ranges

Thanks for pointing this out, this is a manuscript error and we will update it to the accurate labels and ranges.

We hope these changes, along with the new results, analyses and clarifications above, address the reviewer’s concerns. We would be grateful if the reviewer would reconsider their score in light of them, and we are happy to provide any further clarification.

Thank you again for your thoughtful review. We wanted to gently follow up on our responses. We explained the clinical utility of the work with both qualitative text examples, generalization tests and additional steering experiments under realistic conditions. We also compared the impact of steering using text metadata in the prompt compared to our approach and find that ours is more robust to noise, and additive on the benefits seen with textual steering. We also compared SAE-based scores with raw directions, a logistic probe and text-only baselines for predicting clinician preference and found that the our approach is more robust. We also confirmed that the ICD-9 mismatch in Appendix C is confined to the manuscript.

We would be happy to discuss anything further. If these responses address your concerns, we would appreciate it if you would consider updating your score.

Comments on the paper

1 comment

Dear Reviewers,

Thank you again for the time and care you have put into reviewing our paper. We have posted detailed responses to each review. As the discussion period comes to a close, we wanted to gently check whether these responses address your concerns.

In brief, we added several new results. We report the LEACE erasure experiments, which were left out of the appendix by a compilation error. We show that erasing the chapter concept drops diagnosis accuracy from 54.8% to 8.1%, demonstrating the causal use of the concept centers. We also added text-based baselines for better contextualization of the findings. We also demonstrated transfer to a different clinical dataset (MTSamples) without refitting, compared SAE scores against raw directions and probes for predicting clinician preference, and added an off-target control for the generation curves. We also clarified the relation to prior work and the details of our data handling.

If anything is still unclear, we would be glad to discuss it further. If our responses resolve your concerns, we would be grateful if you would consider revisiting your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.