acceptodds
Under review as a conference paper at ICLR 2027

InvExplain: An Agentic Interpretation Framework for SAE Features via Training-Free Diffusion Inversion

Abstract

Large language models (LLMs) have achieved remarkable capabilities, yet their internal representations remain difficult to understand. Sparse autoencoders (SAEs) offer a promising approach by decomposing these representations into more interpretable features. However, automatically interpreting individual SAE features remains challenging. Existing automated pipelines primarily infer explanations from corpus-retrieved activating samples, limiting their evidence to activation patterns represented in a fixed corpus. Feature inversion can expand evidence beyond the corpus, but learned inverse models require additional training and remain tied to their training distributions. To overcome these limitations, we introduce InvExplain, an agentic interpretation framework driven by training-free diffusion posterior sampling (DPS). By leveraging a frozen diffusion language model prior, InvExplain actively acquires new, activation-verified evidence beyond the retrieval corpus without additional training. An LLM agent combines generated and corpus-retrieved evidence to formulate and test competing hypotheses, uses DPS to repair failed positive tests, and refines the explanation. Theoretically, we characterize the prior-dependent distortion of training-based inversion and show that diffusion posterior sampling can be viewed as approximating an activation-constrained distribution under a pretrained language prior. Across three models, InvExplain outperforms corpus-reliant baselines on a shared test of whether a feature activates on a text and, if so, where its activation peaks.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Sparse Autoencoder, Diffusion Language Model, Concept Discovery

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2ZSdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issue where conventional Sparse Autoencoder feature interpretation relies on highly activating samples from a fixed corpus, potentially leading to incomplete or biased explanations. To resolve this, the authors propose InvExplain, which leverages a pretrained Diffusion Language Model and training-free Diffusion Posterior Sampling to generate new activating samples, allowing an LLM agent to verify and refine hypotheses based on both existing and newly generated evidence. In experiments targeting 110 features across three LLMs, InvExplain achieved higher activation and position prediction accuracy compared to existing methods.

Critical strengths
  1. Clearly highlights the limitations of existing corpus-based feature interpretation. It effectively explains the issue of incomplete or biased interpretations caused by relying on a fixed corpus, and intuitively presents examples in Table 2 that uncover activation patterns overlooked by conventional approaches.

  2. The integration of feature inversion and agentic interpretation is compelling. Rather than stopping at generating new activating samples, it utilizes them for hypothesis testing and contrastive testing to demonstrate various activation conditions and the interpretability of features.

  3. Acquires new activation evidence without training a separate inverse model. It proposes a training-free inversion approach leveraging a pretrained DLM, and explains the distributional bias issue of learning-based inversion through theoretical analysis and toy experiments.

Critical weaknesses
  1. Lack of experimental validation proving the necessity of DPS, the core methodology. The main contribution of this paper is Feature Inversion using Training-Free DPS, but the authors do not sufficiently verify whether DPS is superior to simpler generation methods in actual language generation environments. For example, there is no comparison with Best-of-N or Rejection Sampling, where multiple sentences are generated using a general LLM or Unguided DLM, followed by measuring SAE activations to select only high-activating samples. Therefore, it remains unclear whether the performance improvement stems from the effect of DPS itself or simply from acquiring additional activating samples.
  2. The evaluation methodology does not sufficiently demonstrate the generalization performance of actual feature interpretation. The Challenge Set used for evaluation consists of High/Low sentences generated by an LLM based on the feature explanations produced by each method. While evaluating all three methods on the same Challenge Set is an advantage, because the evaluation sentences themselves are generated from the target explanations, this does not fully demonstrate generalization performance on independent natural language distributions. Furthermore, in the absence of ground truth for the meaning of SAE features, Activation and Localization Accuracy merely measure the predictive power of the explanations and do not directly guarantee that the semantic or causal interpretation of the features is correct.
  3. The connection between theoretical analysis and practical DPS implementation is not sufficiently verified. Theorem 2 shows that sampling from the target distribution is possible when using the exact guidance value, but the actual implementation uses approximation processes such as EntRGi-based candidate search, likelihood filtering, and hard selection. Additionally, in the toy experiment of Figure 1, DPS uses an analytically known prior, whereas the learning-based model is trained on finite data, meaning the information conditions of the two methods are not identical. Therefore, it is unclear to what extent the theoretical results and toy experiment support the superiority of DPS in real-world language feature inversion.
Questions for authors
  1. Is DPS actually more effective than standard generation and filtering methods? Can it be compared against baseline approaches that collect activating samples via standard LLM prompting, unguided DLM, or Best-of-NN sampling, and then feed them to the same agent?
  2. How much does each component of InvExplain contribute to the final performance? Are there ablation results isolating the use of corpus evidence alone, the addition of initial DPS-generated evidence, and the incorporation of DPS rewrite?
  3. How are the quality and diversity of the generated evidence validated? Is it possible to quantitatively evaluate metrics such as the actual activation success rate of text generated by DPS, the discovery rate of novel activation patterns distinct from the existing corpus, and semantic or syntactic diversity?
  4. To what extent does the actual DPS generation process approximate the theoretically defined target distribution? The practical implementation uses EntRGi-based candidate search instead of exact posterior sampling. Can you analyze the impact of this approximation on the generated distribution, or quantitatively demonstrate the changes in activation intensity and generation diversity before and after applying guidance?
10 comments
AuthorsOpenReview ↗

Response To Reviewer 2ZSd Part 1

We sincerely thank the reviewer for the thoughtful comments and questions. The changes are highlighted in blue in the revised manuscript. Our point-by-point responses follow.

W1: Lack of experimental validation proving the necessity of DPS, the core methodology. The main contribution of this paper is Feature Inversion using Training-Free DPS, but the authors do not sufficiently verify whether DPS is superior to simpler generation methods in actual language generation environments. For example, there is no comparison with Best-of-N or Rejection Sampling, where multiple sentences are generated using a general LLM or Unguided DLM, followed by measuring SAE activations to select only high-activating samples. Therefore, it remains unclear whether the performance improvement stems from the effect of DPS itself or simply from acquiring additional activating samples.

Q1: Is DPS actually more effective than standard generation and filtering methods? Can it be compared against baseline approaches that collect activating samples via standard LLM prompting, unguided DLM, or Best-of- sampling, and then feed them to the same agent?

Response: Thank you for raising this point. We examined how efficiently DPS collects activating texts and how its use affects the final explanations.

(1) Evidence acquisition. We compare neutral-only DPS with unguided rejection sampling and Best-of-4 using the same pretrained iLLaDA model to measure how quickly each method acquires activation-verified evidence (Appendix M, Table 24).

We evaluate 10 fixed Qwen3-1.7B layer-10 features, aiming to obtain two distinct, activation-verified texts per feature under the same 60-minute budget. Unsuccessful runs are counted as 60 minutes.

MethodMean CandidatesMean Runtime
DPS (Ours, neutral only)3.711.53 min
Unguided Rejection Sampling944.423.17 min
Unguided Best-of-41,020.8031.22 min

Rejection sampling and Best-of-4 took about 2.0 and 2.7 times as long as DPS, respectively. DPS also used fewer completed-text candidates.

(2) Final explanation accuracy. We evaluate the combined contribution of initial DPS evidence, its associated rescue, and failed-High rewriting on 20 Llama-3.2-1B features from layers 10 and 15. Both configurations use the same frozen challenge sets (Appendix H.2, Table 15).

ConfigurationActivation Acc.High localizationJoint Acc.
Without DPS71.67%73.78%69.33%
With DPS generation and rewriting73.17%74.52%69.67%

DPS collected activation-verified evidence faster and with fewer candidates than either unguided baseline. In the ablation, enabling DPS evidence generation, rescue, and rewriting together yielded modest gains in activation accuracy, peak localization, and joint accuracy.

AuthorsOpenReview ↗

Response To Reviewer 2ZSd Part 2

W2.A: The evaluation methodology does not sufficiently demonstrate the generalization performance of actual feature interpretation. The Challenge Set used for evaluation consists of High/Low sentences generated by an LLM based on the feature explanations produced by each method. While evaluating all three methods on the same Challenge Set is an advantage, because the evaluation sentences themselves are generated from the target explanations, this does not fully demonstrate generalization performance on independent natural language distributions.

Response: We thank the reviewer for raising these important concerns. We acknowledge that the original shared challenge set, although evaluated using frozen explanations and independently measured SAE activations, remains conditioned on the explanations being evaluated.

We evaluate the same frozen explanations on naturally occurring passages selected from an external corpus using measured activations. For each feature, all methods are scored on the same ten measured High and ten measured Low nine-token windows (Appendix J, Table 19).

ModelInvExplain JointSAGE JointNeuronpedia-style Joint
Llama-3.2-1B77.88%62.38%70.25%
Llama-3.2-3B81.88%65.12%73.25%
Qwen3-1.7B75.17%73.17%72.17%

InvExplain achieves the highest joint accuracy for each model on this activation-stratified sample of naturally occurring texts.

AuthorsOpenReview ↗

Response To Reviewer 2ZSd Part 3

W2.B. Furthermore, in the absence of ground truth for the meaning of SAE features, Activation and Localization Accuracy merely measure the predictive power of the explanations and do not directly guarantee that the semantic or causal interpretation of the features is correct.

Response: Thank you for raising this point. Activation accuracy and High localization measure whether the frozen explanations predict measured SAE activations and their peak locations. These are behavioral tests of the explanations. Establishing a causal account of feature function would require separate intervention experiments.

Prior work uses SAE features for data filtering and synthesis, as discussed in Appendix A.1. Testing whether more accurate explanations improve those downstream applications is a separate empirical question.

Reviewer_QMAvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work uses a frozen diffusion language model to generate texts guided by target feature activation. Verified texts are combined with retrieved examples to support hypothesis formation, testing, and revision, with inversion also used to rewrite failed positive tests. Experiments cover 110 transcoder features from three models and evaluate activation prediction and peak localization on shared anonymous challenge sets. The method outperforms SAGE and a single pass explanation baseline, with case studies showing how generated evidence can broaden or correct an explanation's boundary.

Critical strengths

The most convincing aspect is that generated evidence changes the substance of the explanations. Table 2 broadens male reference to possessiveness, while Appendix G.1 extends a solvability explanation across polarities. These are concrete, testable changes to the proposed activation boundary. They show why inversion can contribute more than additional wording or examples of an already known pattern.

Critical weaknesses

My main concerns are whether the gains generalize beyond the explanations used to construct the test, how much improvement comes specifically from diffusion guidance, and how stable the results are across features and runs.

  1. Evaluation coverage remains conditioned on the explanations being evaluated. In Appendix G, all challenge texts are generated from the three final explanations and then pooled. Shared inputs, anonymization, and measured activation labels reduce potential bias, but activation modes missed by all three explanations may never enter the test. The results therefore establish prediction quality within the proposed hypothesis coverage more directly than discovery of omitted modes.

  2. The contribution specific to diffusion guidance is not isolated. Table 3 varies inversion budgets and guidance settings, but does not compare against ordinary LLM rewriting with activation verification, or unguided DLM sampling followed by activation filtering, under a matched budget. SAGE provides a useful baseline without inversion, but the comparison does not separate the benefit of diffusion guidance from additional evidence and computation.

  3. Uncertainty across features and runs is not established. Table 1 reports pooled accuracy without confidence intervals across features or repeated runs of the complete explanation pipeline. The 3,300 challenge texts are grouped within 110 features and should not be treated as independent experimental units. I would like to see paired confidence intervals obtained by resampling features, alongside repeated runs on a representative subset.

Questions for authors

Appendix G credits any character overlap with the reference peak token, while Appendix H permits predicted spans of up to 32 characters. Are the predicted span length distributions comparable across methods? Using the existing predictions, could you report localization after controlling for span length while keeping the High/Low decisions fixed? This would help determine whether the improvement reflects more precise peak prediction or is partly explained by broader spans. I do not assume the latter occurs, but the current metric does not distinguish these possibilities.

7 comments
AuthorsOpenReview ↗

Response To Reviewer QMAv Part 1

We sincerely thank the reviewer for the thoughtful comments and questions. The changes are highlighted in blue in the revised manuscript. Our point-by-point responses follow.

W1: Evaluation coverage remains conditioned on the explanations being evaluated. In Appendix G, all challenge texts are generated from the three final explanations and then pooled. Shared inputs, anonymization, and measured activation labels reduce potential bias, but activation modes missed by all three explanations may never enter the test. The results therefore establish prediction quality within the proposed hypothesis coverage more directly than discovery of omitted modes.

Response: Thank you for raising this point. We evaluate the frozen explanations on naturally occurring passages selected from an external corpus using measured feature activations.

For each of the 110 features, we select ten measured High and ten measured Low nine-token windows. High examples are selected within the P25–P75 range of the feature’s positive main-test activations, widened to P10–P90 when needed, and Low examples have zero measured activation. All methods are scored on the same texts (Appendix J, Table 19).

ModelInvExplain JointSAGE JointNeuronpedia-style Joint
Llama-3.2-1B77.88%62.38%70.25%
Llama-3.2-3B81.88%65.12%73.25%
Qwen3-1.7B75.17%73.17%72.17%

InvExplain achieves the highest joint accuracy for each of the three models on this activation-stratified corpus evaluation.

AuthorsOpenReview ↗

Response To Reviewer QMAv Part 2

W2: The contribution specific to diffusion guidance is not isolated. Table 3 varies inversion budgets and guidance settings, but does not compare against ordinary LLM rewriting with activation verification, or unguided DLM sampling followed by activation filtering, under a matched budget. SAGE provides a useful baseline without inversion, but the comparison does not separate the benefit of diffusion guidance from additional evidence and computation.

Response: Thank you for raising this point. We measure the effect of guidance on evidence collection by comparing neutral-only DPS with unguided rejection sampling and Best-of-4 using the same pretrained iLLaDA model (Appendix M, Table 24).

We evaluate 10 fixed SAE features from layer 10 of Qwen3-1.7B, with a common objective of obtaining two distinct, activation-verified texts per feature. All methods share a 60-minute budget per feature, with unsuccessful runs counted as 60 minutes.

MethodMean CandidatesMean Runtime
DPS (Ours, neutral only)3.711.53 min
Unguided Rejection Sampling944.423.17 min
Unguided Best-of-41,020.8031.22 min

Rejection sampling and Best-of-4 took about 2.0 and 2.7 times as long as DPS, respectively. DPS also used fewer completed-text candidates.

AuthorsOpenReview ↗

Response To Reviewer QMAv Part 3

W3: Uncertainty across features and runs is not established. Table 1 reports pooled accuracy without confidence intervals across features or repeated runs of the complete explanation pipeline. The 3,300 challenge texts are grouped within 110 features and should not be treated as independent experimental units. I would like to see paired confidence intervals obtained by resampling features, alongside repeated runs on a representative subset.

Response: We thank the reviewer for this suggestion. We conduct additional analyses to quantify both feature-level statistical uncertainty and run-to-run variability.

(1) Feature-level uncertainty. We draw 100,000 paired bootstrap replicates from the 110 features, resampling within each of the 11 model–layer groups while preserving feature-level clustering and method pairing. Table 20 in Appendix K reports differences in percentage points and 95% confidence intervals.

ComparisonActivationHigh localizationJoint
InvExplain − SAGE+12.33 [9.61, 15.15]+34.07 [26.92, 41.21]+21.94 [18.24, 25.64]
InvExplain − Neuronpedia-style+21.06 [17.94, 24.21]+17.15 [12.35, 22.00]+19.55 [16.52, 22.61]

All intervals for the pooled gains exclude zero under feature resampling of the frozen main-experiment results.

(2) Run-to-run variability. We additionally repeat the complete explanation and evaluation pipeline three times on 10 Qwen3-1.7B layer-20 features. Within each run, all methods are evaluated on the same shared challenge set. Table 21 in Appendix K reports mean ± sample SD (%).

MethodActivation AccuracyHigh localizationJoint Accuracy
InvExplain78.44 ± 3.2064.27 ± 4.6774.67 ± 4.33
SAGE62.67 ± 2.3344.37 ± 7.2259.67 ± 2.52
Neuronpedia-style59.44 ± 0.5141.30 ± 6.8256.33 ± 2.33

InvExplain maintains the highest average accuracy across all three metrics over repeated runs, supporting the robustness of the observed improvements. These additional statistical analyses and experimental details are included in Appendix K.

Reviewer_2LVKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes InvExplain, an agentic framework for automatically interpreting sparse autoencoder (SAE) and transcoder features in large language models (LLMs) using training-free diffusion inversion. The authors identify a key limitation of existing interpretation methods: they primarily rely on high-activation examples retrieved from a fixed corpus, which may provide incomplete or biased evidence about a feature's activation behavior. In order to solve this, they formulate feature inversion as approximate sampling from an exponentially tilted natural-language prior, leveraging a frozen diffusion language model to generate new texts that activate a target SAE feature without training an additional inverse model. The authors conduct experiments on three language models, Llama-3.2-1B, Llama-3.2-3B, and Qwen3-1.7B, across two language model families. They evaluate explanation quality through activation prediction and activation-peak localization on shared challenge sets, which show that the proposed method outperforms SOTA methods.

Critical strengths
  • This paper proposes a novel application of training-free diffusion inversion to SAE interpretation
  • The authors provide strong theoretical motivation and analysis
  • The proposed method has strong empirical improvements over existing baselines
Critical weaknesses
  • Training-based baselines are missing for evaluation
  • There may be bias in the evaluation. The test set is constructed by the methods' own explanations, rather than an independently sampled or externally defined one.
  • There may be some gap between the theoretical analysis and the implemented algorithm. The implementation replaces exact conditional expectations with argmax, applies gradient optimization, and performs greedy selection, which are different from the ideal situation in Theorem 2.
Questions for authors
  • What is the specific contribution of DPS to the overall performance improvement?
  • How does the evaluation ensure that the improvements reflect generalizable explanations rather than performance on explanation-derived examples?
  • What is the computational cost of InvExplain, and how does it scale to large numbers of SAE features?
8 comments
AuthorsOpenReview ↗

Response To Reviewer 2LVK Part 1

We sincerely thank the reviewer for the thoughtful comments and questions. The changes are highlighted in blue in the revised manuscript. Our point-by-point responses follow.

W1: Training-based baselines are missing for evaluation

Response: Thank you for raising this point. We adapt the conditioning architecture of InverseScope [1] to generate evidence for individual SAE features. We fine-tune Qwen3-4B-Base with LoRA, conditioning the generator on SAE activation vectors from layer 10 of Qwen3-1.7B. We refer to this adaptation as InverseScope-LoRA.

We evaluate both methods on 10 fixed layer-10 features of Qwen3-1.7B using the same agentic interpretation framework (Appendix I, Table 18). For each feature, this comparison uses a shared 20-text challenge set drawn from the two frozen explanations.

MethodActivation Acc.High localizationJoint Acc.
InverseScope-LoRA78.50%80.39%77.00%
InvExplain (Ours)83.50%97.56%83.50%

InvExplain improves joint accuracy by 6.50 percentage points without training a dedicated inverse generator.

[1] Luo et al., InverseScope: Scalable Activation Inversion for Interpreting Large Language Models, arXiv:2506.07406v3.

AuthorsOpenReview ↗

Response To Reviewer 2LVK Part 2

W2: There may be bias in the evaluation. The test set is constructed by the methods' own explanations, rather than an independently sampled or externally defined one.

Q2: How does the evaluation ensure that the improvements reflect generalizable explanations rather than performance on explanation-derived examples?

Response: Thank you for this question. We evaluate the frozen explanations on naturally occurring passages selected from an external corpus using measured feature activations.

For each of the 110 features, we select ten measured High and ten measured Low nine-token windows. High examples are selected within the P25–P75 range of the feature’s positive main-test activations, widened to P10–P90 when needed, and Low examples have zero measured activation. All methods are scored on the same texts (Appendix J, Table 19).

MethodActivation AccuracyHigh localizationJoint Accuracy
InvExplain81.64%89.47%78.59%
SAGE75.55%62.91%66.32%
Neuronpedia-style75.18%83.12%71.86%

InvExplain achieves the highest joint accuracy for each of the three models on this activation-stratified corpus evaluation.

AuthorsOpenReview ↗

Response To Reviewer 2LVK Part 3

W3: There may be some gap between the theoretical analysis and the implemented algorithm. The implementation replaces exact conditional expectations with argmax, applies gradient optimization, and performs greedy selection, which are different from the ideal situation in Theorem 2.

Response: Thank you for raising this point. Our original toy experiment considered continuous sampling. Adapting DPS to discrete token sequences introduces approximations in the practical search. To quantify the resulting gap from the target in Theorem 2, we added a toy experiment in a discrete setting closer to language generation.

We use a joint prior pp over two-token sequences (i,j)∈0,…,162(i,j)\in\\{0,\ldots,16\\}^2, define an activation score s(i,j)s(i,j), and compute the ideal target q(i,j)∝p(i,j)eβp(0.75)s(i,j)q(i,j)\propto p(i,j)e^{\beta_p(0.75)s(i,j)} from Theorem 2. At an achieved mean activation of 0.740780.74078 (target 0.750.75), practical DPS yields an empirical total variation (TV) distance of 0.06470.0647 to the target, compared with 0.39670.3967 for the prior. Direct target sampling with the same 18,000 samples yields a median empirical TV of 0.04650.0465. The gap of 0.01820.0182 shows that practical DPS comes close to direct target sampling on this metric in the discrete toy. Figure 2 in Section 2.4 visualizes the prior, activation scores, exact target, and empirical DPS distribution.

DistributionMean activationTV to target
Reference prior pp (exact)0.042120.3967
Target qq (exact)0.750
Direct target samples (18,000)—0.0465
Practical algorithm (18,000)0.740780.0647

The detailed results and experimental settings are provided in Appendix F, including Table 10.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief Summary of the Discussion Phase

Dear AC and Reviewers,

We sincerely thank you for the time and care you have devoted to reviewing our submission and for your constructive feedback. We recognize the additional workload and challenges of this review cycle and appreciate your continued efforts. To assist the AC in preparing the meta-review, we summarize the main concerns and our analyses below.

Reviewer 2LVK raised questions about training-based baselines, generalization to independent text, and how closely the practical DPS sampler approximates its theoretical target. In response, we compared DPS with a trained inversion baseline, evaluated frozen explanations on an external corpus, and tested the sampler in a setting where the target distribution could be computed exactly. After reading our responses, the reviewer confirmed that all concerns had been addressed and explained their decision to retain a score of 3: “Given that the max score is 4, I have decided to maintain my score.”

Reviewer QMAv asked whether the gains generalize beyond explanation-generated texts, how stable they are across features and runs, and how much diffusion guidance contributes. We added an independent-corpus evaluation, feature-level confidence intervals, repeated runs, and a comparison of generators using the same interpretation agent. The reviewer acknowledged that DPS collects activating examples faster but observed that unguided sampling matched or exceeded its downstream activation prediction accuracy. They maintained their score of 2.

Reviewer 2ZSd asked how DPS compares with simpler sampling methods and whether its contribution to interpretation accuracy could be isolated. The reviewer welcomed the independent-corpus evaluation and the new efficiency measurements. In our follow-up comparison, rejection sampling matched DPS on activation prediction accuracy, while Best-of-4 performed slightly better. The reviewer acknowledged the efficiency advantage of DPS but remained unconvinced of a DPS-specific accuracy gain and maintained their score of 2.

In our rebuttal, we emphasize that InvExplain is an interpretation framework that combines corpus examples with generated, activation-verified evidence to test and refine feature explanations. Across all four generators tested, InvExplain outperformed both SAGE and the Neuronpedia-style baseline on activation prediction accuracy. DPS is one way to generate this evidence, and in our benchmark it collected activating examples faster than both rejection sampling and Best-of-4.

Finally, we sincerely thank all reviewers for their constructive comments and the AC for their efforts throughout the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.