acceptodds
Under review as a conference paper at ICLR 2027

Keep Features Dense and Explanations Sparse: Adaptive Interpretability at Test Time

Abstract

Simple data-derived linear directions can expose highly interpretable structure in neural representations, yet their activations are typically signed and dense. Sparse autoencoders instead learn non-negative sparse features, coupling feature learning with a sparsity constraint. We ask whether sparsity is needed in the representation at all. More broadly, test-time computation remains comparatively underexplored in interpretability. We introduce **TTS-AE**, which learns a dense, non-negative reconstructive representation from data-grounded directions and introduces sparsity only when an explanation is requested. At test time, *Test-Time Sparsification* (TTS) selects a compact, input-specific subset of features and infers the remaining activations from the covariance structure of the dense representation until a desired reconstruction fidelity is reached. Across three model–dataset pairs, the resulting dense features are more interpretable than nonlinear sparse autoencoder features, while TTS reaches matched fidelity with substantially fewer positively activated concepts than sparse autoencoders and sparse approximation baselines. The same dense concept space also supports covariance-aware interventions and sparse target-specific global explanations. Our results suggest that interpretability and explanation sparsity need not be learned jointly: dense representations can support sparse explanations constructed adaptively through additional computation at test time.

Primary area:
interpretability and explainable AI
Keywords:
Sparse autoencoders, SAEs, interpretability, mechanistic interpretability, explainable AI, dictionary learning, geometry-grounded representations, vision interpretability

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_qxuTOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a test time interpretability method. For an intermediate activation the method learns a k-means initialised autoencoder, encouraged not to move too far away from that initialisation, applies post-hoc VLM derived descriptions of each learned autoencoder feature based on maximally activating examples, and then for a particular input selects features greedily until their covariance aware latent variable reconstruction (Eq 3) gives an activation reconstruction that meets some pre-determined fidelity level. To validate this they evaluate this method on a single layer from each of three models on three matched datasets, measuring R2R^2 reconstruction vs avg interpretability (mixture of three common VLM interpretability scores) showing their method KMeansAE to have higher interpretability for a given level of reconstruction than other methods. They also show they achieve higher R2R^2 for a given level of sparsity than other methods. Lastly there are some experiments showing the validity of steering using their reconstructed activations.

Critical strengths
  1. I thought, for the most part, that this paper was well written and clearly laid out, aside from a few issues with how the method is described, I found the motivation, intuition behind the model and explanatory purpose of each experiment easy to understand. Given the recent interest in matched pursuit for interpretability and the limits of SAEs in identifying meaningful features this work seems timely and like it would be of interest to the field.

  2. The breadth and quality of experiments is high, with a few caveats. Generally I thought the experiments were well thought out, informative and chosen thoughtfully.

Critical weaknesses
  1. I would have liked a bit more qualitative evaluation of features, or some examples of features extracted from each method here since automated feature quality metrics can sometimes be deceiving. Figure 6 was a good start but some side by side comparisons might be enlightening here.

  2. How fair is your metrics panel? You exclude the monosemanticity measures despite this being the very measure your method doesn't perform well on and also that I would expect a method which depends on feature covariance to struggle with. Isn't monosemanticity of features desirable? How sure are you of the validity of the other 3 metrics more generally, do they correlate with human judgement?

Questions for authors
  1. How are the ranks computed in equation 8 calculated?
  2. In the OpenImages/CLIP RN50 panel in figure 4 your method seems to plateau while Vanilla and top-k SAE are still improving. This seems fine because you achieve higher levels of interpretability (according to the metrics you select). Why then do you push R2R^2 so high in figure 12 and report vanilla SAE having higher sparsity?
  3. Could you explain the refitting step of your method in a bit more detail? It is mentioned at several points but doesn't appear to be explained.
2 comments

We thank the reviewer for the suggestions on qualitative evaluation, metric choice, and the steering protocol.

W1 — Side-by-side feature examples. Following the reviewer's suggestion, we added three comparisons between TTS and Vanilla SAE on the same CLIP ViT-B/32 inputs, with spatial responses, labels, and top-activating images (Appendix H.4, Figures 16–18). We also ran a grounding experiment to test whether the positive concepts shown in an explanation describe objects or scenes that are present in the input image. At comparable reconstruction fidelity, Grounding DINO gives TTS the highest mean score across all three model–dataset pairs and both description methods (Section 4.2, Table 1, and Appendix H.3. See H7qq W2).

W2 — Monosemanticity and human agreement. We agree with the reviewer that monosemanticity belongs in the main comparison. We originally reported MS separately because it gave very similar scores to standard linear extractors such as PCA, NMF, and the identity baseline. The revised average includes MS and adds SemanticLens monosemanticity. KMeansAE still scores higher on the five-metric average than the evaluated Vanilla and TopK SAEs at high reconstruction fidelity, and some SAE configurations score higher on MS (Section 4.1, Figure 3, and Appendices G.3–G.4).

Clarity measures coherence among top-activating images, INVERT measures agreement with annotated classes, and CLIP-Dissect measures alignment with text labels. Each captures a different aspect of feature quality, and their scales differ. Prior studies compare MS and SemanticLens polysemanticity with human judgments in their own settings. How well the original three metrics and the five-metric average agree with human judgments on our dictionaries remains an open question. MS study · SemanticLens study

Q1 — Ranks in Eq. (8). We sort class scores from highest to lowest and record the target's position, starting at one. The scores are classifier logits for ResNet-18 and DenseNet-161, and image–text cosine similarities for CLIP RN50. Ties follow the sort order. Cases that start at rank one are excluded from the rank-gap metric, and none occurred in the reported sweep. Section 4.3 and Appendix I.1 give the candidate classes, CLIP prompt, and sorting details.

Q2 — The CLIP plateau and high-fidelity comparisons. TTS has a clear advantage when the goal is to show as few positive concepts as possible at comparable reconstruction fidelity. Figure 4 shows this advantage on CLIP RN50 in the regime with few positive activations. At higher fidelity, some SAE configurations continue to improve as the TTS curve levels off. We show that range to make clear where the advantage narrows. We read the reference to Figure 12 as referring to the layerwise analysis, now Figure 13 in Appendix G.9. That analysis uses CLIP ViT-B/32 and compares reconstruction and activation counts across layers. The KMeansAE counts there are for the dense encoder before test-time selection. Vanilla SAE is sparser at that stage. Figure 4 addresses the separate question of how many positive concepts TTS displays after selection.

Q3 — Refitting. We thank the reviewer for pointing out that our wording suggested a separate refitting step. We revised the description to clarify the reconstruction procedure. TTS holds the original selected activations fixed, including zeros, and updates the covariance reconstruction in Eq. (3) as it adds coordinates. Figure 2 displays two positive concepts and uses all 617 selected observations for reconstruction. We clarified this in Section 3, Figure 2, and the discussion of Algorithm 1.

Thank you for taking the time to address my concerns. I believe the changes made improve the paper and as such I will raise my score.

Reviewer_kZPGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TTS-AE, an interpretability framework that separates learning an interpretable feature representation from enforcing sparsity. TTS-AE first learns a dense, non-negative concept representation initialized from KMeans-derived directions and regularized to preserve the geometry of the original representation space. At inference time, Test-Time Sparsification (TTS) greedily selects an input-specific subset of concepts and uses the covariance structure of the dense latent space to reconstruct unselected features until a desired reconstruction fidelity is reached. The same concept space is also used for covariance-aware interventions and sparse global explanations. The method is evaluated across multiple vision model-dataset pairs using reconstruction and semantic interpretability metrics, comparisons with sparse autoencoders and sparse approximation baselines, and intervention experiments. The results suggest that dense concept representations can retain strong semantic structure while supporting compact, adaptive explanations at test time.

Critical strengths

The paper presents a clear and conceptually interesting reframing of sparsity in interpretability by separating feature learning from explanation sparsity. Rather than enforcing sparsity during representation learning, TTS-AE learns a dense, geometry-grounded concept space and constructs input-specific sparse explanations at test time. This is a meaningful perspective because it challenges the common assumption that interpretable representations themselves must be sparse and provides a flexible way to trade explanation complexity against reconstruction fidelity.

The empirical evaluation is broad and relatively careful. The paper evaluates semantic quality, reconstruction fidelity, sparse explanation construction, and concept interventions across multiple model-dataset pairs, and includes comparisons against several sparse autoencoder and sparse reconstruction baselines. The covariance-aware intervention results are particularly useful because they go beyond purely semantic or reconstruction-based evaluation and provide evidence that the learned concepts are functionally connected to model behavior.

Critical weaknesses

The paper’s central sparsity comparison is difficult to interpret as an apples-to-apples comparison of explanation complexity. TTS defines explanation sparsity using the final positively weighted support after refitting, even though reconstruction may depend on a much larger greedy support and covariance-based inference of the full dense latent state. For example, Figure 2 selects 617 coordinates during search but counts only two as active after refitting. This makes the headline comparison against SAE active-feature counts potentially misleading, since the methods achieve reconstruction using representations with substantially different inferential and computational mechanisms. This issue directly affects the paper’s central claim that dense representations can yield substantially sparser explanations.

The experiments do not fully disentangle the effects of geometry-grounded initialization and alignment from the absence of a sparsity constraint when explaining KMeansAE's semantic-quality advantage over SAE baselines. While the results demonstrate that interpretable features can be learned without enforcing sparsity, they do not establish which design choice is primarily responsible for the observed improvement over the evaluated SAEs. A geometry-matched sparse baseline would help isolate the contribution of decoupling feature learning from sparsity.

Questions for authors
  1. The main sparsity results count only positively weighted features after refitting, while TTS may select many more coordinates during its greedy search. Could you clarify why positive support is the appropriate quantity for comparison with SAE active-feature counts, and whether the main conclusions remain when nonzero or total support is used instead?

  2. How can the effect of removing representation sparsity be disentangled from the effect of KMeans initialization and geometry alignment? In particular, could the semantic advantage of KMeansAE over the evaluated SAEs primarily arise from preserving the KMeans geometry rather than from learning a dense representation?

  3. Section 3 defines the explanation in terms of the selected support and its activations, while the main sparsity comparison uses positive coefficients after refitting. Could the authors clarify how these two definitions relate, and whether the covariance-mediated reconstruction dependencies should be considered when assessing explanation complexity?

1 comment

We thank the reviewer for raising the support comparison and proposing a control for geometry and sparsity.

W1 — Fairness of the sparsity comparison. We revised the claim to make the distinction explicit. TTS separates the number of positively expressed concepts a reader inspects from the number of observations used to reconstruct the underlying representation. This lets TTS present compact, input-specific concept summaries at a chosen reconstruction fidelity, with a larger reconstruction support and additional computation as explicit costs. Figure 2 shows two such concepts, and the reported fidelity comes from covariance reconstruction using all 617 selected observations. In the new grounding experiment, the displayed TTS concepts score higher than the evaluated MP and SAE baselines for both description sources on the retained fidelity-matched subset of each dataset (Section 4.2, Table 1, and Appendix H.3, see H7qq W2). The new runtime benchmark shows the cost of this trade-off: about 1,222 selected coordinates and 5 seconds per image at the shared target (Appendix H.5, Table 2, see H7qq Q2).

W2 — Geometry-matched sparsity control. Following the reviewer's suggestion, we ran a new experiment that varies the activation L1L_1 and alignment penalties in KMeansAE on CLIP RN50/OpenImages while holding the architecture and KMeans initialization fixed (Section 4.1 and Appendix G.7, Figure 10). Stronger L1L_1 shrank activations without consistently reducing the active count, and at strong alignment the five-metric average fell from about 0.60 to 0.52 at the strongest penalty. One possible explanation is the bias-free ReLU encoder: the penalty can shrink positive activations, while the encoder has no independent bias to move each feature's activation threshold. The sweep tests the penalty at the achieved fidelities. It shows that KMeansAE can learn interpretable concepts without an explicit sparsity penalty.

Five-metric average interpretability on CLIP RN50/OpenImages (Appendix G.7, Figure 10). Rows vary the alignment penalty and columns vary the activation L1L_1 penalty.

λalign\lambda_{\mathrm{align}} vs. λsparse\lambda_{\mathrm{sparse}}0010−610^{-6}5×10−65\times10^{-6}10−510^{-5}10−410^{-4}10−310^{-3}
10410^{4}0.600.600.580.580.530.52
10310^{3}0.600.580.550.540.530.52
10210^{2}0.600.540.540.540.530.52
10110^{1}0.590.540.540.540.530.52
110.560.540.540.540.530.52
10−110^{-1}0.530.540.540.540.530.52
10−210^{-2}0.520.540.540.540.530.52
10−310^{-3}0.520.540.540.540.530.52

Scores are rounded to two decimal places and reflect each configuration's achieved reconstruction fidelity.

Q1 — Positive, nonzero, and total support. Positive and nonzero counts coincide for TTS at the same threshold. In the new benchmark at approximately R2=0.80R^2=0.80 (Appendix H.5, Table 2, see H7qq Q2), TTS uses 10.16 positive/nonzero activations out of 1,222.27 selected coordinates, while MP uses 159.89 positive/nonzero atoms out of 159.89 selected. The advantage in positive-set size reverses when counting total support. The positive-set count is comparable to the active SAE count as a measure of what is shown. In the three Vanilla SAE comparisons in Appendix H.4 (Figures 16–18, see qxuT W1), TTS displays fewer concepts and uses more selected coordinates than the SAE has active features. The sensitivity sweep makes this trade-off adjustable. Raising the TTS penalty from 0 to 1 at the 0.80 target changes the mean positive count from 143.30 to 10.33, total support from 221.27 to 1,239.70, and time from 0.49 to 4.82 s/image. Both settings reached the target on all 64 images (Appendix H.5).

Q2 — Could KMeans geometry explain the semantic advantage? Yes. Over part of the alignment sweep, stronger alignment improves semantic scores while the active count stays roughly stable (Appendix G.6, Figure 9). Together with the new penalty sweep (Appendix G.7, see W2 above), the results support geometry preservation as a contributor and show that an explicit sparsity objective is unnecessary for learning interpretable features in this setting.

Q3 — How do the two sets relate? The positive set is the subset of selected coordinates with positive original encoder activations. Eq. (3) uses the entire total support, including observed zeros, to estimate the full latent vector. TTS recomputes that estimate as the selected set grows, with the observations held fixed. We corrected the reference to refitting in Section 3. The selected observations and the shared covariance model both contribute to reconstruction, so we report total support alongside the size of the displayed positive set.

Reviewer_H7qqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TTS-AE, which learns dense, interpretable concepts and constructs sparse explanations at test time using covariance-based reconstruction. Experiments across three vision benchmarks demonstrate improved semantic quality and more compact explanations than sparse autoencoder baselines, with additional evaluations of concept steering and global explanations.

Critical strengths

S1: Separating dense feature learning from explanation sparsity is well motivated. One representation can support different fidelity targets without retraining.

S2: Across three model–dataset pairs, the dense dictionary achieves better Clarity, INVERT, and CLIP-Dissect scores than the evaluated SAEs at comparable reconstruction. Norm-matched steering with random controls adds useful evidence that the concepts have behavioral relevance.

Critical weaknesses

W1: My main concern is what the sparsity claim counts. Figure 2 reports 617 search coordinates but two active concepts after refitting; Algorithm 1 omits that refit. It remains unclear whether coordinates assigned zero coefficients after refitting can be discarded without affecting reconstruction fidelity. This raises concerns about the fairness of the sparsity comparison with SAEs.

W2: The semantic evaluation primarily measures whole dictionaries. Figure 2 provides useful qualitative evidence, but there is no systematic comparison of the concepts actually selected by TTS and SAEs across datasets at matched fidelity. This leaves the interpretability benefit of the selection procedure uncertain.

Questions for authors

Q1: Can the two final active concepts alone reproduce the reported R2≈0.8R^2 \approx 0.8 in Figure 2? Could the author clarify the refitting procedure and whether additional selected coordinates are needed for reconstruction?

Q2: At matched fidelity, how does TTS compare with MP in runtime and positive, nonzero, and total support using the final checkpoint?

2 comments

We thank the reviewer for asking us to clarify the support counts and evaluate the selected concepts directly.

W1 — What the sparsity comparison counts. We revised Section 3 and Figure 2 to separate the concepts shown to the reader from the observations used to reconstruct the representation. The trained encoder and decoder and the reference mean and covariance are fixed. For each image, TTS keeps the encoder activations fixed and adapts which coordinates it selects, and how many, to the image and the requested fidelity. At comparable fidelity, TTS can show fewer positively activated concepts than an SAE. The full support and runtime quantify the separate cost of reconstructing the representation. Section 4.1 shows that the KMeansAE dictionary used by TTS scores higher on average than the evaluated SAEs in the high-reconstruction regime. The new grounding experiment also finds higher visual grounding scores for the concepts TTS displays (Section 4.2, Table 1, see W2 below).

W2 — Quality of selected concepts. Following the reviewer's suggestion, we ran a new grounding experiment (Section 4.2, Table 1, and Appendix H.3). For each image and method, we sampled up to five positive concepts and scored their CLIP-Dissect and INVERT descriptions against the image with Grounding DINO. We kept images for which all six methods achieved R2∈[0.75,0.85]R^2\in[0.75,0.85] and had at least one scorable concept. The fidelity and label requirements retained 344 ImageNet images (34.4%), 511 Places365 images (51.1%), and 351 OpenImages images (35.1%) of the 1,000 initially evaluated per dataset.

For each SAE family and model, fixed uses one checkpoint chosen on a separate calibration set. Adaptive chooses a checkpoint from the same sweep for each test image, taking the in-band reconstruction closest to R2=0.80R^2=0.80.

Table 1. Visual grounding of selected concepts. Mean ± sample standard deviation across the same retained images for all methods and both description sources within each model. Higher is better.

MethodImageNet CLIP-DissectImageNet INVERTPlaces365 CLIP-DissectPlaces365 INVERTOpenImages CLIP-DissectOpenImages INVERT
TTS0.58 ± 0.230.53 ± 0.250.52 ± 0.180.56 ± 0.200.58 ± 0.200.35 ± 0.26
MP0.47 ± 0.190.45 ± 0.190.38 ± 0.140.46 ± 0.150.50 ± 0.150.27 ± 0.17
Vanilla SAE (fixed)0.47 ± 0.180.42 ± 0.190.39 ± 0.140.47 ± 0.140.52 ± 0.170.27 ± 0.17
Vanilla SAE (adaptive)0.49 ± 0.190.43 ± 0.200.40 ± 0.140.47 ± 0.140.50 ± 0.170.28 ± 0.19
TopK SAE (fixed)0.49 ± 0.180.44 ± 0.190.40 ± 0.140.46 ± 0.150.48 ± 0.150.24 ± 0.15
TopK SAE (adaptive)0.48 ± 0.180.43 ± 0.190.39 ± 0.140.45 ± 0.140.47 ± 0.150.25 ± 0.15

On this retained subset, TTS displayed fewer positive concepts at comparable fidelity, and achieved the highest mean Grounding DINO score for all three datasets and both description methods, leading the strongest baseline by 0.06–0.12 in each column. The three new comparisons with Vanilla SAE show individual examples with labels and spatial responses (Appendix H.4, Figures 16–18, see qxuT W1).

Q1 — Can two positive concepts reproduce Figure 2? No. Following the reviewer's question, we reran the covariance reconstruction for the Figure 2 image using only its two positive concepts. The adjusted feature-space R2R^2 was −1.62-1.62, compared with 0.800.80 using the full support of 617 observations, including 615 observed zeros. The negative score means that the two-coordinate reconstruction has greater squared error than decoding the mean concept vector. For this image, the positive set summarizes the visible concepts, and the selected zeros provide information needed for the reported reconstruction.

In an SAE, a zero coefficient contributes zero to the additive reconstruction. In TTS, a selected zero is an observed value that can influence the covariance estimate. An unselected coordinate is unobserved. We thank the reviewer for catching an error in our description. We removed the phrase “after refitting.” TTS holds the selected activations fixed and recomputes the covariance estimate in Eq. (3) as the selected set grows. We clarified this in Section 3, the Figure 2 caption, and the discussion of Algorithm 1.

Q2 — Runtime and support at comparable fidelity. As the reviewer asked, we reran the TTS–MP benchmark with the final CLIP RN50 KMeansAE checkpoint and a shared R2=0.80R^2=0.80 target. Both methods reached the target on 256 OpenImages images. The trade-off persisted. TTS displayed about 10 positive concepts versus 160 for MP, and used about 1,222 selected coordinates versus 160 distinct MP atoms. Batch-amortized time was about 5.01 s/image for TTS and 12.71 ms/image for MP. Section 4.2 and Appendix H.5 (Table 2) report the counts and timings. See kZPG Q1 for the support comparison.

Thanks for your reply. I have raised my score to 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Comment

We thank the reviewers for their time and effort. We are grateful that reviewers H7qq and kZPG found the separation of feature learning from explanation sparsity well motivated and saw value in the norm-matched steering experiments. We also appreciate that kZPG and qxuT found the evaluation broad and informative, and that qxuT found the motivation and experimental design clear.

We have uploaded a revised manuscript with the following changes.

  1. TTS method clarification. We revised Section 3 and Figure 2 to explain how TTS selects coordinates from a dense concept representation and uses their observed activations to infer the full latent vector from covariance. The total support includes selected zero activations. TTS prioritizes a small set of positively activated concepts for the reader to inspect. It uses a larger set of observations and additional computation to reconstruct the representation at the requested fidelity. In the new benchmark with the final checkpoint, TTS used about 1,222 selected coordinates and took about 5 seconds per image, compared with 160 selected atoms and 12.71 ms per image for MP at a shared target of R2=0.80R^2=0.80 (Section 4.2, “Cost of Adaptivity,” and Appendix H.5, see H7qq Q2).
  2. Concept quality. In a new experiment at comparable reconstruction fidelity, TTS displayed fewer positive concepts and led all six mean grounding comparisons on the retained images (Section 4.2, Table 1, and Appendix H.3, see H7qq W2). We revised the main interpretability average to include MS and SemanticLens monosemanticity and now report all five metrics separately (Section 4.1, Figure 3, and Appendices G.3–G.4, see qxuT W2).
  3. Geometry and sparsity. In a new CLIP RN50/OpenImages experiment, we jointly varied the alignment penalty and an activation L1L_1 penalty within KMeansAE. Adding the sparsity penalty yielded no consistent gain in average interpretability, and the strongest penalty lowered it under strong alignment (Section 4.1 and Appendix G.7, Figure 10, see kZPG W2 and Q2).
  4. Additional examples and clarifications. We added three side-by-side comparisons with Vanilla SAE (Appendix H.4, Figures 16–18, see qxuT W1). We clarified how steering ranks are computed in Eq. (8) (Section 4.3 and Appendix I.1, see qxuT Q1) and added a plot of changes in the target score (Appendix I.1, Figure 20).

We address each reviewer's questions in separate replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.