acceptodds
Under review as a conference paper at ICLR 2027

ExSAL: Explaining Sparse Autoencoder Latents with Domain-Adapted LLMs

13 views

Abstract

Sparse Autoencoders (SAEs) decompose language model activations into interpretable features. The standard approach to describing these features prompts a general-purpose LLM with the feature’s top-activating tokens and asks it to produce a natural-language description. However, when the analyzed model has been domain-adapted, its SAE features encode specialized knowledge that a general-purpose explainer may fail to describe accurately. We argue that the domain-adapted model itself is better positioned to explain its own features. To this end, we introduce EXSAL (Explaining Sparse Autoencoder Latents with Domain-Adapted LLMs), a mechanistic interpretability framework for analyzing features from domain-adapted LLMs. Given a domain-adapted model, EXSAL trains an SAE on its activations, and uses that same model to generate natural-language descriptions of the learned features. The framework is applied to two independently adapted models: an enterprise-software model and a finance model. Our experiments show that EXSAL outperforms the general-purpose LLM explainer and narrows the gap to a far larger frontier model, using only the domain-adapted model already at hand and requiring no external explainer. These observations indicate that EXSAL produces more accurate feature descriptions, providing a stronger foundation for analyzing SAE latents in domain-adapted LLMs without requiring a separate explainer model.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Mechanistic Interpretability, Domain Adaptation, Sparse Autoencoders

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_43hoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Standard sparse autoencoder interpretability pipelines typically delegate feature naming to general-purpose LLM explainers. However, when probing domain-adapted models whose SAE features capture specialized knowledge, general-purpose explainers often yield inaccurate descriptions. To address this, the authors investigate whether a domain-adapted model is better suited to interpret its own features. They introduce ExSAL, a framework that uses the probed domain-adapted LLM as a self-explainer to generate natural-language descriptions of its extracted SAE features. Evaluating across enterprise-software and financial domains, the authors demonstrate that ExSAL outperforms general-purpose LLM explainers while narrowing the performance gap relative to significantly larger frontier models.

Critical strengths
  • Novel conceptual perspective: The paper challenges standard practice in SAE interpretability by showing that domain-adapted models can act as their own explainers. This provides a cost-effective alternative that achieves competitive interpretation quality without relying on expensive frontier models.
  • Methodological rigor: The research questions are well-structured and logically connected, with downstream inquiries (e.g., RQ4 and RQ5) building naturally upon preceding empirical findings.
  • Clear structure and presentation: The manuscript is well-organized, articulates its core rationale clearly, and incorporates helpful visual diagrams that guide the reader through the experimental setup and contributions.
Critical weaknesses
  • Marginal empirical gains: The performance gains achieved by switching from a base model to a domain-adapted explainer appear modest. In Table 1, ExSAL yields only minor improvements over the baseline (e.g., Detection AUROC increases from ~0.81 to 0.82, Simulation score from 0.29 to 0.32, and fuzzy accuracy improves by just 0.03 over the uninformative blank baseline in Figure 3). Similarly, Table 2 indicates that in the financial domain, Detection AUROC increases by only 0.02 (0.92 to 0.94), while fuzzy accuracy actually falls below the base model. The authors should tone down claims of performance superiority.
Questions for authors
  1. What are the specific implications of training the SAE on the final layer? Could extracting features from intermediate layers yield different findings, and might layer selection depend on the domain or the specific latent features being targeted?
  2. Could the authors provide concrete examples of extracted latent features along with their corresponding input text and generated descriptions?
9 comments

We thank the reviewer for their evaluation of our work, and we address each of the mentioned concerns in detail below.

W1: Marginal gains, tone down claims of superiority

"Gains from base → domain-adapted explainer appear modest. Table 1: Detection AUROC ~0.81 → 0.82, Simulation 0.29 → 0.32 […]. Table 2 (finance): Detection AUROC +0.02 (0.92 → 0.94). […] In the financial domain, fuzzy accuracy actually falls below the base model. The authors should tone down claims of performance superiority."

We agree that the absolute gains sometimes seem small (for example Detection AUROC from 0.92 to 0.94 in the finance domain). However, it is important to note that in relative terms the gains are quite high. Simulation Score increases by about 9 % (0.290 to 0.317) in the enterprise domain and by about 7 % (0.426 to 0.456) in the finance domain, and the step from 0.923 to 0.935 Detection AUROC in the finance domain removes about 16 % of the remaining error. In absolute terms the last percentage points are also the hardest to gain.

Due to our setup, the numbers are also reliable. For each feature we sample different quantiles, generate several descriptions and let the judge score each of them several times. The confidence intervals within a feature are small (about ±0.004 to ±0.007 in Tables 1 and 2), so the results of a feature stay stable even when the sampled tokens, the descriptions and the judge runs change. For us this is a clear sign that the improvements we measure are not noise.

In addition to the three LLM-judge metrics, we also did a human evaluation. We took the 50 curated FinTrain latents and showed, for each one, the six passages where it fires most strongly, with the activating token highlighted and its sentence as context. Below the passages were two descriptions, one from the BASE explainer and one from ExSAL, and the task was to pick the description that fits better to the highlighted tokens. We distributed these among the three authors, and none of us knew which model wrote which description. Which model wrote description A or B was random for each item (ExSAL was A in 25 of 50 items), so the position cannot give away the label.

ExSAL was preferred in 39 of 50 cases, and each annotator preferred it on a significant majority of their assigned items. It won both as first option A (18/25) and as second option B (21/25). These results are consistent with the metrics in the paper.

For the final version of the paper we will add this human evaluation, to show next to the gains in the metrics that humans also prefer the ExSAL descriptions. Testing whether the effects grow with larger models is future work.

Q1: Training the SAE on the final layer

"What are the implications of training the SAE on the final layer? Could extracting features from intermediate layers yield different findings, and might layer selection depend on the domain or the specific latent features being targeted?"

We admit that we should have tested more than one layer, ideally an early, a middle and a late one. Due to compute constraints we could only train SAEs for one layer per domain and took the last layer. Testing other layers was outside the scope of this work, so we cannot say how our findings change there.

There is evidence that the choice of the layer has an influence on the latents that are found. Templeton et al. [1] train on the middle residual stream of Claude 3 Sonnet because they expect "interesting, abstract features" there, and Gao et al. [2] choose a layer near the end of the network, which "should contain many features without being specialized for next-token predictions". Gao et al. also report that the reconstruction quality of the SAEs changes with the layer. Gurnee et al. [3] show for neurons that the late layers of a model contain groups of neurons with a clear role for the output tokens, which differ from the neurons in earlier layers. Together this suggests that the layer can make a difference for which latents are learned and how they can be explained, and it is plausible that this might also depend on the domain and on the targeted latents.

For the camera-ready version we will name the choice of the last layer more clearly in the limitations section, discuss the influence the layer can have on the latents, and list testing other layers (early, middle and late) as future work.

References [1] Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet (Templeton et al., 2024) [2] Scaling and evaluating sparse autoencoders (Gao et al., 2024)
[3] Universal Neurons in GPT2 Language Models (Gurnee et al., 2024)

Q2: Concrete examples of latents

"Could the authors provide concrete examples of extracted latent features along with their corresponding input text and generated descriptions?"

At the request of the reviewer, we provide the complete input, output and pipeline for two latents in the following tables. As a short reminder, the pipeline works as follows.

  1. All tokens on which the latent fires in the test split are collected.
  2. The tokens are ranked by activation.
  3. The ranked tokens are split into five quantile bands.
  4. The explainer sees 20 tokens per band, each with its context of ±10 tokens, and writes a description.
  5. The judge sees the description and the held-out tokens (5 per band and 5 tokens with zero activation, each with its context) and computes the score. The details of this step depend on the metric, so they are different for Simulation Score, Detection AUROC and Fuzzing Accuracy.

Below we provide these tables for two latents of the Finance domain (public FinTrain text, Llama-Fin-8B SAE, layer 31). We only show some of the token examples, because it would take up too much space otherwise. The activating tokens are in bold.

Reviewer_SFMdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ExSAL, a framework for generating more accurate natural-language explanations of SAE features in domain-adapted language models. The authors argue that general-purpose LLM explainers may struggle to interpret specialized concepts learned through domain adaptation, whereas the domain-adapted models themselves possess relevant knowledge. ExSAL trains a JumpReLU SAE on model activations, identifies domain-relevant latent features, and uses the domain-adapted LLM to generate feature descriptions from activating contexts.

Critical strengths
  • It is a practical motivation that general-purpose LLMs may struggle to interpret specialized concepts encoded in domain-adapted models. Using the domain-adapted model itself as an explainer is a simple and practical approach that leverages knowledge already acquired during domain adaptation.

  • The paper evaluates ExSAL across two distinct domains and different model families, comparing it against base-model explainers, RAG-augmented variants, and a larger frontier model. The results show consistent improvements in Simulation Score and Detection AUROC over the corresponding base models.

Critical weaknesses
  • The novelty is quite limited. The proposed ExSAL framework primarily replaces a general-purpose LLM explainer with its domain-adapted counterpart, while relying on existing SAE architectures, feature-selection strategies, and explanation-evaluation metrics. Prior work has already explored self-explanation of SAE features [1], as well as more advanced methods involving activation-based feedback and iterative explanation refinement. This weakens its methodological contribution relative to the novelty expected at ICLR.

  • The evaluation scope is also limited, and validation of Explanation Faithfulness is insufficient. The experiments evaluate only 50 carefully selected domain-relevant features per domain, raising concerns about generalizability to broader SAE feature distributions. Moreover, explanation quality is assessed primarily through LLM-based activation prediction metrics, which do not necessarily establish whether the descriptions faithfully reflect the model's internal mechanisms. Although the authors evaluate with an alternative judge, independent human evaluation or causal intervention-based validation is missing. Additionally, improvements are relatively modest and inconsistent across metrics, particularly Fuzzing Accuracy in the finance domain. These limitations weaken the evidence that ExSAL provides substantially more accurate and reliable feature interpretations rather than simply generating more domain-specific descriptions.

[1] Haozhe Chen, Carl Vondrick, and Chengzhi Mao. SelfIE: self-interpretation of large language model embeddings. ICML 2024.

Questions for authors
  • The proposed ExSAL framework primarily uses a domain-adapted LLM as the explainer instead of a general-purpose model. Given that prior work has already explored self-explanation of SAE features, could the authors clarify the key methodological contributions beyond substituting a domain-adapted model for the base explainer, what is the novel methodological contribution?

  • The proposed evaluation relies on LLM-based activation prediction metrics, which may not necessarily reflect whether the generated descriptions faithfully capture the causal roles of SAE features. Could the authors provide independent validation through human experts or activation intervention experiments? How can the evaluation distinguish genuinely more accurate feature interpretations from descriptions that simply contain more domain-specific terminology?

  • Could the authors isolate the effects of continued pre-training and instruction fine-tuning, rather than attributing the observed improvements entirely to domain-specific knowledge?

6 comments

We thank the reviewer for reviewing our work and providing meaningful feedback. We address each of the reviewer's questions/concerns in detail below.

W1 / Q1: Novelty beyond substituting the explainer

W1: "ExSAL primarily replaces a general-purpose explainer with its domain-adapted counterpart, while relying on existing SAE architectures, feature-selection strategies, and evaluation metrics. Prior work has explored self-explanation of SAE features [1 — SelfIE], and activation-based feedback / iterative refinement."
Q1: "Given prior work on self-explanation of SAE features, could the authors clarify the novel methodological contribution beyond substituting a domain-adapted model for the base explainer?"

We respond to this concern in three different dimensions:

1.1 What the paper contributes

We have kept the SAE architecture, the feature selection, and the metrics standard on purpose. We only change the parts of the pipeline that were necessary, so that our results are comparable to other work and so that the explainer is the only thing that differs between the configurations. The contribution is therefore not a new architecture but instead a framework that answers the question of whether an explainer needs to know the domain of the features it describes, which to the best of our knowledge has not been tested before. Practically, the work would be useful to inspect the latents of a domain-adapted model that performs better than its general-purpose parent.

1.2 Difference between SelfIE and ExSAL

We agree that the previous work SelfIE [1] is related, and we already cite it in Section 4.2. Like ExSAL, it is a method to explain what the internals of a model represent, but it differs from our approach in two ways: (i) it works differently: it replaces a placeholder token in an interpretation prompt with the hidden embedding that should be interpreted and lets the same model describe it, whereas the explainer in ExSAL reads the top-activating contexts as text. (ii) SelfIE does not look at the domain knowledge of the explainer. It is evaluated on LLaMA-2-70B-Chat, without domain-adapted models, and it does not compare explainers with and without domain knowledge.

As SelfIE is a different approach and comes with its own limitations, some of which are mentioned by [2], who apply it to SAE features. Comparing it directly with our approach would be a good direction for future work. It would also be possible to run our comparison with and without domain knowledge for [1]: when explaining embeddings, one could use different models for the explanation instead of the same model, and investigate whether the domain knowledge of the explainer makes a difference. We will explicitly state this as future work.

1.3 Activation-based feedback and iterative refinement

Methods of this kind, such as the explainer-critic loop of [3], the interpretability agents of [4] and the activation-reward training of [5], improve the procedure with which a description is produced. Whereas ExSAL experiments with who produces the descriptions instead. But these works are indeed related, and thus we already cite them in Section 4.2.

W2 / Q2: Limited evaluation scope, validation of faithfulness, and domain terminology

W2: "The experiments evaluated only 50 carefully selected domain-relevant features per domain, raising concerns about generalizability to broader SAE feature distributions. Moreover, explanation quality is assessed primarily through LLM-based activation prediction metrics, which do not necessarily establish faithfulness to internal mechanisms. While the authors include an alternative judge, independent human evaluation or causal-intervention validation is missing."
Q2: "Could the authors provide independent validation via human experts or activation-intervention experiments? How can the evaluation distinguish genuinely more accurate feature interpretations from descriptions that simply contain more domain-specific terminology?"

2.1 Why only 50 selected features per domain

Why do we choose features instead of using random ones? Selecting good domain-relevant features is already a hard task on its own, and this is also the reason why we cannot simply use random ones: not every feature is a domain feature that carries useful knowledge. As described in Appendix A.8, most random features are polysemantic, syntactic or out of domain. We therefore have to select domain-relevant features.

Why only 50 features? This is a trade-off under limited compute. In the explanation step, the explainer sees different tokens depending on which tokens get sampled from the quantiles. Large language models are also not deterministic, so the same input can lead to different descriptions. In the judging step, an LLM is involved as well, so it can score the same description differently.

To take this into account, we work with a set of 50 features and build exactly this behavior into our evaluation: for each feature we sample three times from the quantiles, generate three descriptions per sample and let the judge score each description three times (3 x 3 x 3 = 27 scores per feature and metric). This setup lets us compute meaningful confidence intervals inside of a feature, and it shows that our results (when fixing a feature) stay consistent even when different tokens are sampled, different descriptions are generated, and the judge is called several times. We preferred this over having more features, so that we can be confident the results are consistent for each feature. We admit that a higher number of features would further strengthen the evidence for our findings. We will therefore add this point to the limitations section.

2.2 Use Human Evaluation as Extension to our Main-Metrics

To address the missing human evaluation, we performed a human evaluation of 50 features. We took the 50 curated FinTrain latents and showed, for each one, the six passages where it fires most strongly, with the activating token highlighted and its sentence as context. Below the passages were two descriptions, one from the BASE explainer and one from ExSAL, and the task was to pick the description that fits better to the highlighted tokens. We distributed these to three domain-expert human annotators to label them. Which model wrote description A or B was random for each item (ExSAL was A in 25 of 50 items), so the position cannot give away the label.

ExSAL was preferred in 39 of 50 cases, and each annotator preferred it on a significant majority of their assigned items. It won both as first option A (18/25) and as second option B (21/25). These results are consistent with the findings in our paper and show that Exsal outperforms the Base model.

2.3 Accurate descriptions or just more domain-specific terminology?

To test this, we ran a keyword-chain experiment. For each of the 50 features, we took the description of the domain-adapted explainer (ExSAL) and let a powerful frontier model (Claude-4.7-Opus) extract only the content keywords, which we then chained together with spaces. This keeps the domain terms but removes the sentences, so the relations between the terms are lost. The chains have a median of 11 keywords (Finance) and 12 keywords (enterprise), compared to about 57 words in the original descriptions. We scored the keyword chains with the same judge and the same held-out tokens as the original description. We report mean scores with 95% CI over the 50 features below (θ_domain judge):

Finance

DescriptionSimulationDetection AUROCFuzzing
Blank (empty description)0.105 ± 0.0520.509 ± 0.0500.700 ± 0.028
Keyword chain (this new experiment)0.420 ± 0.0500.930 ± 0.0220.734 ± 0.032
BASE0.422 ± 0.0460.938 ± 0.0180.768 ± 0.028
ExSAL0.453 ± 0.0490.943 ± 0.0180.734 ± 0.029

Enterprise

DescriptionSimulationDetection AUROCFuzzing
Blank (empty description)0.074 ± 0.0570.594 ± 0.0440.560 ± 0.039
Keyword chain (this new experiment)0.274 ± 0.0520.782 ± 0.0470.523 ± 0.038
BASE0.273 ± 0.0530.808 ± 0.0420.560 ± 0.045
ExSAL0.285 ± 0.0490.813 ± 0.0420.574 ± 0.038

To further keep the comparison fair, before a description exists, there are two sampling steps in our pipeline: we sample from the quantiles, and then we sample the descriptions. For this experiment, we derive the keyword chain from one fixed description, so all rows use the same configuration: the first quantile sample, the first description and the same held-out tokens. Only the third level of the branching remains: the repeated judge calls (each description is scored three times by the judge). This is a design choice, as with other descriptions, the ExSAL description could contain different keywords. ExSAL is the description the keyword chain was built from for these experiments, and BASE is the description of the BASE explainer in exactly this configuration.

As expected, the keyword chain performs better than the empty description in most cases and even dominates the BASE description in one case, but it does not reach the level of the ExSAL description. This separates the description being accurate vs. just domain terminology: even though the keywords carry some of the domain knowledge, the relations between them are not described. A chain of keywords does not say how they belong together. This supports the idea that simply chaining domain terms is not enough, and that a real understanding of these terms is needed to write a good description of a latent.

Reviewer_BR3HOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The proposed ExSAL uses a domain-adapted model (enterprise software; finance) describe its own latents, picking 50 latents that a frontier chatbot rates as specific and domain-related, and then comparing description quality across explainers. Quality is measured by three existing judge-based scores of how well a description lets a model predict latent activations on unused contexts. The adapted explainer wins most comparisons over its parent, retrieval hurts in the first domain and helps in the second, and the frontier model stays ahead throughout.

Critical strengths
  • The motivation for local domain-specific SAE deployment and interpretation is sensible and tangible.
Critical weaknesses
  • (W1) The quality dimensions ("ExSAL outperforms the general-purpose LLM explainer", l. 023; "better SAE feature descriptions", l. 087) remain very vague. The metrics are quite contested in literature that is not cited here.
    • The measures ask a judge model to use a description as instructions for guessing activations. They say nothing about whether the described concept matches what the feature does. Huang et al. (2023) [1] find substantial error rates with weak causal effect. Paulo & Belrose (2025) [2] argue for evaluating latents while skipping descriptions altogether.
  • (W2) The RQs seem to be derived post-hoc and mirror the experimental setup. This should obviously be the other way around. More RQs doesn't mean more rigor or substance.
    • Section 4.2's claim that existing methods do not look at domain adaptation at all is too broad: work on a clinical multimodal models on exactly this kind of setting exists in Bouzid et al (2025) [3].
    • Input-based descriptions miss what a feature does to the model's outputs, an axis where writing descriptions from the model's changed behaviour works better (Gur-Arieh et al., 2025) [4].
  • (W3) Nothing in the experimental design tests the privileged access aspect or the initial motivation of not needing a separate explainer, e.g., a third model that knows the domain or a setting where the adapted model described features of the parent model's activations.
    • Also, statistical reporting makes the small differences look firmer than they actually are, and the fitted SAEs themselves are not characterized. Each table cell comes from a few dozen features, judged on a handful of token contexts. The CIs seem to come from resampling repeated generation and judging calls over the same items, so they describe how determinstic a description generator is, but variation across features or evaluation items is neglected, since every configuration was run on the same few features.

[1] Rigorously Assessing Natural Language Explanations of Neurons (Huang et al., 2023)
[2] Evaluating SAE interpretability without explanations (Paulo & Belrose, 2025)
[3] Insights into a radiology-specialised multimodal large language model with sparse autoencoders (Bouzid et al., 2025)
[4] Enhancing Automated Interpretability with Output-Centric Feature Descriptions (Gur-Arieh et al., 2025)

Questions for authors
  • (Q1) Could you justify the choice of quality metrics which lets you say ExSAL beats the vanilla model?
  • (Q2) Can you add the two crossings from (W3): an unrelated domain-adapted model writing descriptions for these latents, and your adapted explainer describing features taken from the parent model's SAE?
13 comments

We thank the reviewer for the thorough evaluation of our work, and we address each of the mentioned concerns in detail below.

W1 / Q1: Choice of quality metrics

W1: "The quality dimensions […] remain very vague. The metrics are quite contested […]. The measures ask a judge model to use a description as instructions for guessing activations […]. Huang et al. (2023) find substantial error rates with weak causal effect. Paulo & Belrose (2025) argue for evaluating latents while skipping descriptions altogether."
Q1: "Could you justify the choice of quality metrics which lets you say ExSAL beats the vanilla model?"

1.1 Justification of quality metrics

We thank the reviewer for pointing this out, but we would like to clarify that if we had reported only the Simulation score, that would have been a weak basis for saying that ExSAL beats the vanilla model. Instead, the paper reports three metrics: Simulation score, Detection AUROC and Fuzzing accuracy.

For the Simulation score, we do see the criticism. But we still decided to report it, as it is an established metric in the field and in order to make our findings comparable to other work.

Detection and Fuzzing are different in an important way. Both were introduced by [2] specifically for SAE latents, in a paper that came out about a year after [1]. Among the scoring methods they evaluate, Fuzzing, Simulation, and Detection have the highest Spearman correlation with human scores (0.69, 0.60 and 0.59), and they are also very cost-efficient to run. Since then, these metrics have been used in several other works in the interpretability field. The purpose of reporting several metrics instead of one is to avoid depending on a single contested score.

On the argument that one could evaluate latents "while skipping descriptions altogether" [3], in our perception, this is only a reasonable choice if descriptions are not of interest. However, there are still cases e.g. when auditing a model, where researchers are interested in natural language descriptions of a latent for debugging purposes.

1.2 Human evaluation

In addition to these arguments, we also did a human evaluation of 50 features. We took the 50 curated FinTrain latents and showed, for each one, the six passages where it fires most strongly, with the activating token highlighted and its sentence as context. Below the passages were two descriptions, one from the BASE explainer and one from ExSAL, and the task was to pick the description that fits better to the highlighted tokens. We provided these to three human domain-expert annotators to label them. Which model wrote description A or B was random for each item (ExSAL was A in 25 of 50 items), so the position cannot leak the labels.

ExSAL was preferred in 39 of 50 cases, and each annotator preferred it on a significant majority of the assigned annotation samples. It won both when shown first as A (18/25) and when shown second as B (21/25). These results are consistent with the findings in our paper and show that Exsal outperforms the Base model.

W2: Research questions, domain-adaptation claim, input- vs. output-centric descriptions

W2(a) The RQs seem post-hoc

"The RQs seem to be derived post-hoc and mirror the experimental setup; more RQs doesn't mean more rigor."

We agree with the reviewer's point that more RQs do not mean more rigor. However, firstly, we would like to clarify that the primary purpose of posing multiple research questions is to present the findings more clearly to the reader. Secondly, all of the RQs follow from one hypothesis that is stated in the introduction section: knowledge acquired during domain adaptation helps a model describe domain-specific features. RQ1 tests this hypothesis directly. Each of the other RQs then tests one alternative explanation for a positive RQ1 result:

  • RQ2: does the same domain knowledge work equally well in context (RAG over the identical corpus) instead of in the weights?
  • RQ3: can scale replace domain knowledge (frontier explainer)?
  • RQ4: is the gain only self-preference of the judge (independent judge)?
  • RQ5: is the effect specific to one domain and model family (second domain)?

W2(b) Over-broad domain-adaptation claim

"Section 4.2's claim that existing methods do not look at domain adaptation at all is too broad — Bouzid et al. (2025) […]"

We thank the reviewer for mentioning the work [4], which is about how sparse autoencoders are used in domain-adapted settings. We commit to adding it to Section 4.3, where we already discuss [5] and [6].

However, the sentence in Section 4.2 is about something more specific: that existing approaches "do not consider how domain adaptation affects explanation quality". [4] train a Matryoshka SAE on the radiology model MAIRA-2, but all feature descriptions are generated by a general-purpose explainer. They do not compare explainers, and they do not use the domain-specialised model to explain its own features, which is what ExSAL does.

W2(c) Input- vs. output-centric descriptions

"Input-based descriptions miss what a feature does to the model's outputs; output-centric descriptions (Gur-Arieh et al., 2025) work better on that axis."

We thank the reviewer for this suggestion. We performed the output-based evaluation of [7] on the finance domain. Each latent is steered to a fixed KL divergence (±0.25, ±0.5). The judge sees a description and three sets of generated text, one from the described latent and two from random latents, and has to pick the right set (chance is 33.3%). We used all 50 latents with 9 descriptions per explainer and added an empty-description control (DUMMY).

Output-based accuracy (%)BASEExSALFRONTIERDUMMY
all 50 latents84.989.189.179.6
16 latents without repetition77.182.682.653.5

We observe that ExSAL beats BASE and matches the frontier model, so the gain we see on the input side also shows up on the output side.

When looking at the DUMMY values, you will notice immediately that they are very high (79.6%), although we would expect them to be close to chance. The reason is that steering many of our latents (34 of 50, compared to 1 of 20 random distractor latents) makes the model repeat a single word over and over. This makes the task easy for the judge: even without any meaningful description, it can just pick the set that repeats itself and be correct. The previous work [7] samples targets and distractors at random and does not compare against an empty description, and thus this problem does not come up in their setting.

However, after excluding the latents that repeat, we are left with 16 latents. On those, DUMMY drops to a much lower 53.5%, and the order of the explainers stays the same: ExSAL and FRONTIER have the same value (82.6%), and BASE is lower (77.1%).

We will add this output-centric evaluation with the thorough explanation to the paper.

W3(a) / Q2: Is the "privileged access" aspect tested?

W3: "Nothing in the design tests the privileged-access aspect […], e.g. a third model that knows the domain, or the adapted model describing features of the parent model's activations."
Q2: "Can you add (i) an unrelated domain-adapted model writing descriptions for these latents, and (ii) your adapted explainer describing features taken from the parent model's SAE?"

From the provided review description, we understood the reviewer's request as replacing the explainer with a different model that also has the domain knowledge but is not the model whose latents are explained. We performed this experiment for the Finance domain.

The best choice would be to train an additional model on exactly the same data as the FinTrain model. Due to time and compute constraints, this was not feasible. Instead, we took AdaptLLM/finance-chat [8], as it fulfilled the following constraints:

  • it is domain-adapted for finance by continued pre-training
  • it is a different model (Llama-2-7B-Chat instead of Llama-3-8B) but with comparable size
  • has open weights
  • it is a chat model, so it can follow the instructions in the system prompt and the user prompt

We executed the experiment exactly as described in the paper: same 50 latents, same held-out tokens, same judges, with and without RAG. Mean scores with 95% CI over features (θ_domain judge):

ExplainerSimulationDetection AUROCFuzzing
DUMMY (empty description)0.111 [0.080, 0.141]0.496 [0.462, 0.530]0.697 [0.678, 0.715]
finance-chat (unrelated, domain-adapted)0.394 [0.364, 0.423]0.903 [0.885, 0.919]0.721 [0.705, 0.735]
BASE (Llama-3-8B-Instruct)0.426 [0.398, 0.455]0.923 [0.908, 0.937]0.747 [0.732, 0.761]
ExSAL (Llama-Fin-8B)0.456 [0.424, 0.486]0.935 [0.921, 0.949]0.734 [0.718, 0.749]
FRONTIER0.471 [0.442, 0.498]0.954 [0.942, 0.965]0.749 [0.733, 0.764]

The unrelated domain-adapted model writes useful descriptions, far above the empty description on every metric, but it scores lower than BASE and ExSAL. So domain knowledge in a different model does not necessarily reproduce ExSAL's advantage. This is what we would expect, because finance-chat was adapted on different data than the model whose latents it describes. It is hard to know how much of this gap comes from this and how much from base model strength (finance-chat is Llama-2-based, BASE is Llama-3), so we do not claim that this experiment isolates privileged access. It does show that a domain-adapted model is not automatically a good explainer for another model's latents.

We will add a separate section in the paper to illustrate this ablation.


W3(b): Statistical reporting: source of the CIs

"CIs appear to resample repeated generation/judging calls over the same items — they capture generator determinism, not variation across features or eval items […]."

We thank the reviewer and agree on what our CIs measure. The CIs in the paper resample the records within each feature. They show how stable the score of one explainer is for a fixed latent. For a fixed latent, the score can still change for three reasons: (i) which quantiles we sample decides which activating tokens the explainer sees, so we can get different descriptions. (ii) the explainer LLM is non-deterministic, so the same input can lead to different descriptions. (iii) the same holds for the judge: each description is scored three times, and the judge can give different scores for the same description. We considered it important to show that our measurements are stable under these sources of variation.

We now report CIs over features. The mean values are identical to the ones in the paper; only the intervals change (θ_domain judge, mean ± 95% CI over the 50 latents):

DomainExplainerSimulationDetection AUROCFuzzing
EnterpriseBASE0.290 ± 0.0430.809 ± 0.0280.552 ± 0.033
ExSAL0.318 ± 0.0410.822 ± 0.0260.569 ± 0.032
FRONTIER0.351 ± 0.0400.863 ± 0.0220.578 ± 0.028
FinanceBASE0.426 ± 0.0290.923 ± 0.0140.747 ± 0.015
ExSAL0.456 ± 0.0310.935 ± 0.0140.734 ± 0.015
FRONTIER0.471 ± 0.0280.954 ± 0.0110.749 ± 0.016

As expected, the CIs over features are larger than the within-feature CIs. Our intuition is that this is because the features themselves are very different and differ in how hard they are to explain. Some features are easy to describe, others encode a more complex matter. Even among the 50 features, some probably encode more syntax and others more semantic content. This is a property of a feature, not of the explainer. We will add this table to the paper for the final version.

Comments on the paper

1 comment

We thank all reviewers for their detailed engagement. Below is a brief summary of the additions made during the rebuttal and the changes committed to the final version of the paper.

Claims and evaluation measures (R1, R3). We will tie every comparative claim to the property it measures, with a new table in Section 3, and replace "more accurate feature descriptions" by the measured results. We will state that Fuzzing Accuracy does not improve in the finance domain, and report the confidence intervals over features next to the intervals within a feature.

Human evaluation (R1, R2, R3). We added a pairwise human evaluation on 50 finance latents (ExSAL preferred in 39 of 50 cases). For the final version, every annotator will label every description so that we can report agreement, and the evaluation will cover both domains and tokens from different quantiles. We will also add a human version of the output-centric task, following Gur-Arieh et al. (2025).

Output-centric evaluation (R1). We added the output-centric evaluation, with results per description and per latent for BASE, ExSAL and FRONTIER, and we checked the pipeline for the identical ExSAL and FRONTIER accuracy in finance and found no error. We specify the repetition criterion and will additionally report a relaxed criterion that keeps 27 instead of 16 latents as a sensitivity analysis.

Privileged access and alternative explanations (R1, R2). We added an experiment with an unrelated domain-adapted model, SAE statistics for both domains, and a keyword-chain experiment that tests whether domain terminology alone explains the gain. For the final version, we will add an IFT-only control to separate continued pre-training from instruction fine-tuning, and we will soften the mechanism claims in Section 2 and Appendix A.6.1.

Research questions and related work (R1, R2). We will restructure the research questions around one central question, label the additional analyses as exploratory, add Bouzid et al. to Section 4.3, and add the differences to SelfIE and to iterative explainer methods to the related work.

Layer choice, evaluation scope and examples (R2, R3). We will state the choice of the last layer and the use of 50 selected features per domain as limitations and add concrete latent examples with the full pipeline (input tokens per quantile, held-out tokens, descriptions and scores) to the appendix.

We hope these additions address the main concerns, and we kindly ask the reviewers to reconsider their scores in light of this new information.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.