- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Standard sparse autoencoder interpretability pipelines typically delegate feature naming to general-purpose LLM explainers. However, when probing domain-adapted models whose SAE features capture specialized knowledge, general-purpose explainers often yield inaccurate descriptions. To address this, the authors investigate whether a domain-adapted model is better suited to interpret its own features. They introduce ExSAL, a framework that uses the probed domain-adapted LLM as a self-explainer to generate natural-language descriptions of its extracted SAE features. Evaluating across enterprise-software and financial domains, the authors demonstrate that ExSAL outperforms general-purpose LLM explainers while narrowing the performance gap relative to significantly larger frontier models.
Critical strengths
- Novel conceptual perspective: The paper challenges standard practice in SAE interpretability by showing that domain-adapted models can act as their own explainers. This provides a cost-effective alternative that achieves competitive interpretation quality without relying on expensive frontier models.
- Methodological rigor: The research questions are well-structured and logically connected, with downstream inquiries (e.g., RQ4 and RQ5) building naturally upon preceding empirical findings.
- Clear structure and presentation: The manuscript is well-organized, articulates its core rationale clearly, and incorporates helpful visual diagrams that guide the reader through the experimental setup and contributions.
Critical weaknesses
- Marginal empirical gains: The performance gains achieved by switching from a base model to a domain-adapted explainer appear modest. In Table 1, ExSAL yields only minor improvements over the baseline (e.g., Detection AUROC increases from ~0.81 to 0.82, Simulation score from 0.29 to 0.32, and fuzzy accuracy improves by just 0.03 over the uninformative blank baseline in Figure 3). Similarly, Table 2 indicates that in the financial domain, Detection AUROC increases by only 0.02 (0.92 to 0.94), while fuzzy accuracy actually falls below the base model. The authors should tone down claims of performance superiority.
Questions for authors
- What are the specific implications of training the SAE on the final layer? Could extracting features from intermediate layers yield different findings, and might layer selection depend on the domain or the specific latent features being targeted?
- Could the authors provide concrete examples of extracted latent features along with their corresponding input text and generated descriptions?
9 comments
Q1: Training the SAE on the final layer
"What are the implications of training the SAE on the final layer? Could extracting features from intermediate layers yield different findings, and might layer selection depend on the domain or the specific latent features being targeted?"
We admit that we should have tested more than one layer, ideally an early, a middle and a late one. Due to compute constraints we could only train SAEs for one layer per domain and took the last layer. Testing other layers was outside the scope of this work, so we cannot say how our findings change there.
There is evidence that the choice of the layer has an influence on the latents that are found. Templeton et al. [1] train on the middle residual stream of Claude 3 Sonnet because they expect "interesting, abstract features" there, and Gao et al. [2] choose a layer near the end of the network, which "should contain many features without being specialized for next-token predictions". Gao et al. also report that the reconstruction quality of the SAEs changes with the layer. Gurnee et al. [3] show for neurons that the late layers of a model contain groups of neurons with a clear role for the output tokens, which differ from the neurons in earlier layers. Together this suggests that the layer can make a difference for which latents are learned and how they can be explained, and it is plausible that this might also depend on the domain and on the targeted latents.
For the camera-ready version we will name the choice of the last layer more clearly in the limitations section, discuss the influence the layer can have on the latents, and list testing other layers (early, middle and late) as future work.
References
[1] Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet (Templeton et al., 2024)
[2] Scaling and evaluating sparse autoencoders (Gao et al., 2024)
[3] Universal Neurons in GPT2 Language Models (Gurnee et al., 2024)
Q2: Concrete examples of latents
"Could the authors provide concrete examples of extracted latent features along with their corresponding input text and generated descriptions?"
At the request of the reviewer, we provide the complete input, output and pipeline for two latents in the following tables. As a short reminder, the pipeline works as follows.
- All tokens on which the latent fires in the test split are collected.
- The tokens are ranked by activation.
- The ranked tokens are split into five quantile bands.
- The explainer sees 20 tokens per band, each with its context of ±10 tokens, and writes a description.
- The judge sees the description and the held-out tokens (5 per band and 5 tokens with zero activation, each with its context) and computes the score. The details of this step depend on the metric, so they are different for Simulation Score, Detection AUROC and Fuzzing Accuracy.
Below we provide these tables for two latents of the Finance domain (public FinTrain text, Llama-Fin-8B SAE, layer 31). We only show some of the token examples, because it would take up too much space otherwise. The activating tokens are in bold.
We thank the reviewer for their evaluation of our work, and we address each of the mentioned concerns in detail below.
W1: Marginal gains, tone down claims of superiority
We agree that the absolute gains sometimes seem small (for example Detection AUROC from 0.92 to 0.94 in the finance domain). However, it is important to note that in relative terms the gains are quite high. Simulation Score increases by about 9 % (0.290 to 0.317) in the enterprise domain and by about 7 % (0.426 to 0.456) in the finance domain, and the step from 0.923 to 0.935 Detection AUROC in the finance domain removes about 16 % of the remaining error. In absolute terms the last percentage points are also the hardest to gain.
Due to our setup, the numbers are also reliable. For each feature we sample different quantiles, generate several descriptions and let the judge score each of them several times. The confidence intervals within a feature are small (about ±0.004 to ±0.007 in Tables 1 and 2), so the results of a feature stay stable even when the sampled tokens, the descriptions and the judge runs change. For us this is a clear sign that the improvements we measure are not noise.
In addition to the three LLM-judge metrics, we also did a human evaluation. We took the 50 curated FinTrain latents and showed, for each one, the six passages where it fires most strongly, with the activating token highlighted and its sentence as context. Below the passages were two descriptions, one from the BASE explainer and one from ExSAL, and the task was to pick the description that fits better to the highlighted tokens. We distributed these among the three authors, and none of us knew which model wrote which description. Which model wrote description A or B was random for each item (ExSAL was A in 25 of 50 items), so the position cannot give away the label.
ExSAL was preferred in 39 of 50 cases, and each annotator preferred it on a significant majority of their assigned items. It won both as first option A (18/25) and as second option B (21/25). These results are consistent with the metrics in the paper.
For the final version of the paper we will add this human evaluation, to show next to the gains in the metrics that humans also prefer the ExSAL descriptions. Testing whether the effects grow with larger models is future work.