- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the issue where conventional Sparse Autoencoder feature interpretation relies on highly activating samples from a fixed corpus, potentially leading to incomplete or biased explanations. To resolve this, the authors propose InvExplain, which leverages a pretrained Diffusion Language Model and training-free Diffusion Posterior Sampling to generate new activating samples, allowing an LLM agent to verify and refine hypotheses based on both existing and newly generated evidence. In experiments targeting 110 features across three LLMs, InvExplain achieved higher activation and position prediction accuracy compared to existing methods.
Critical strengths
-
Clearly highlights the limitations of existing corpus-based feature interpretation. It effectively explains the issue of incomplete or biased interpretations caused by relying on a fixed corpus, and intuitively presents examples in Table 2 that uncover activation patterns overlooked by conventional approaches.
-
The integration of feature inversion and agentic interpretation is compelling. Rather than stopping at generating new activating samples, it utilizes them for hypothesis testing and contrastive testing to demonstrate various activation conditions and the interpretability of features.
-
Acquires new activation evidence without training a separate inverse model. It proposes a training-free inversion approach leveraging a pretrained DLM, and explains the distributional bias issue of learning-based inversion through theoretical analysis and toy experiments.
Critical weaknesses
- Lack of experimental validation proving the necessity of DPS, the core methodology. The main contribution of this paper is Feature Inversion using Training-Free DPS, but the authors do not sufficiently verify whether DPS is superior to simpler generation methods in actual language generation environments. For example, there is no comparison with Best-of-N or Rejection Sampling, where multiple sentences are generated using a general LLM or Unguided DLM, followed by measuring SAE activations to select only high-activating samples. Therefore, it remains unclear whether the performance improvement stems from the effect of DPS itself or simply from acquiring additional activating samples.
- The evaluation methodology does not sufficiently demonstrate the generalization performance of actual feature interpretation. The Challenge Set used for evaluation consists of High/Low sentences generated by an LLM based on the feature explanations produced by each method. While evaluating all three methods on the same Challenge Set is an advantage, because the evaluation sentences themselves are generated from the target explanations, this does not fully demonstrate generalization performance on independent natural language distributions. Furthermore, in the absence of ground truth for the meaning of SAE features, Activation and Localization Accuracy merely measure the predictive power of the explanations and do not directly guarantee that the semantic or causal interpretation of the features is correct.
- The connection between theoretical analysis and practical DPS implementation is not sufficiently verified. Theorem 2 shows that sampling from the target distribution is possible when using the exact guidance value, but the actual implementation uses approximation processes such as EntRGi-based candidate search, likelihood filtering, and hard selection. Additionally, in the toy experiment of Figure 1, DPS uses an analytically known prior, whereas the learning-based model is trained on finite data, meaning the information conditions of the two methods are not identical. Therefore, it is unclear to what extent the theoretical results and toy experiment support the superiority of DPS in real-world language feature inversion.
Questions for authors
- Is DPS actually more effective than standard generation and filtering methods? Can it be compared against baseline approaches that collect activating samples via standard LLM prompting, unguided DLM, or Best-of- sampling, and then feed them to the same agent?
- How much does each component of InvExplain contribute to the final performance? Are there ablation results isolating the use of corpus evidence alone, the addition of initial DPS-generated evidence, and the incorporation of DPS rewrite?
- How are the quality and diversity of the generated evidence validated? Is it possible to quantitatively evaluate metrics such as the actual activation success rate of text generated by DPS, the discovery rate of novel activation patterns distinct from the existing corpus, and semantic or syntactic diversity?
- To what extent does the actual DPS generation process approximate the theoretically defined target distribution? The practical implementation uses EntRGi-based candidate search instead of exact posterior sampling. Can you analyze the impact of this approximation on the generated distribution, or quantitatively demonstrate the changes in activation intensity and generation diversity before and after applying guidance?
10 comments
Response To Reviewer 2ZSd Part 2
W2.A: The evaluation methodology does not sufficiently demonstrate the generalization performance of actual feature interpretation. The Challenge Set used for evaluation consists of High/Low sentences generated by an LLM based on the feature explanations produced by each method. While evaluating all three methods on the same Challenge Set is an advantage, because the evaluation sentences themselves are generated from the target explanations, this does not fully demonstrate generalization performance on independent natural language distributions.
Response: We thank the reviewer for raising these important concerns. We acknowledge that the original shared challenge set, although evaluated using frozen explanations and independently measured SAE activations, remains conditioned on the explanations being evaluated.
We evaluate the same frozen explanations on naturally occurring passages selected from an external corpus using measured activations. For each feature, all methods are scored on the same ten measured High and ten measured Low nine-token windows (Appendix J, Table 19).
| Model | InvExplain Joint | SAGE Joint | Neuronpedia-style Joint |
|---|---|---|---|
| Llama-3.2-1B | 77.88% | 62.38% | 70.25% |
| Llama-3.2-3B | 81.88% | 65.12% | 73.25% |
| Qwen3-1.7B | 75.17% | 73.17% | 72.17% |
InvExplain achieves the highest joint accuracy for each model on this activation-stratified sample of naturally occurring texts.
Response To Reviewer 2ZSd Part 3
W2.B. Furthermore, in the absence of ground truth for the meaning of SAE features, Activation and Localization Accuracy merely measure the predictive power of the explanations and do not directly guarantee that the semantic or causal interpretation of the features is correct.
Response: Thank you for raising this point. Activation accuracy and High localization measure whether the frozen explanations predict measured SAE activations and their peak locations. These are behavioral tests of the explanations. Establishing a causal account of feature function would require separate intervention experiments.
Prior work uses SAE features for data filtering and synthesis, as discussed in Appendix A.1. Testing whether more accurate explanations improve those downstream applications is a separate empirical question.
Response To Reviewer 2ZSd Part 1
We sincerely thank the reviewer for the thoughtful comments and questions. The changes are highlighted in blue in the revised manuscript. Our point-by-point responses follow.
Response: Thank you for raising this point. We examined how efficiently DPS collects activating texts and how its use affects the final explanations.
(1) Evidence acquisition. We compare neutral-only DPS with unguided rejection sampling and Best-of-4 using the same pretrained iLLaDA model to measure how quickly each method acquires activation-verified evidence (Appendix M, Table 24).
We evaluate 10 fixed Qwen3-1.7B layer-10 features, aiming to obtain two distinct, activation-verified texts per feature under the same 60-minute budget. Unsuccessful runs are counted as 60 minutes.
Rejection sampling and Best-of-4 took about 2.0 and 2.7 times as long as DPS, respectively. DPS also used fewer completed-text candidates.
(2) Final explanation accuracy. We evaluate the combined contribution of initial DPS evidence, its associated rescue, and failed-High rewriting on 20 Llama-3.2-1B features from layers 10 and 15. Both configurations use the same frozen challenge sets (Appendix H.2, Table 15).
DPS collected activation-verified evidence faster and with fewer candidates than either unguided baseline. In the ablation, enabling DPS evidence generation, rescue, and rewriting together yielded modest gains in activation accuracy, peak localization, and joint accuracy.