acceptodds
Under review as a conference paper at ICLR 2027

Which Features Matter? Sparse Latent Intervention for Prediction-Preserving Explanations

Abstract

Deep neural networks achieve strong predictive performance across a broad range of domains, yet their internal representations remain largely opaque: for a given prediction, it is generally unclear which internal features the model relies on, and whether they correspond to meaningful structure in the underlying domain. Sparse autoencoders (SAEs) decompose hidden representations into interpretable features, but reveal only what a layer represents, not which features a prediction actually depends on, while intervention-based methods typically assess features one at a time and overlook their interactions. We present Sparse Latent Intervention (SLI), a mechanistic interpretability framework that identifies, for a frozen model and a single input, a sparse set of SAE features that closely reproduces the model’s prediction. SLI casts this inverse problem as a sparse selection problem: stochastic gates over the entire feature dictionary are jointly optimised under an expected- penalty to preserve the prediction, while the counterfactual representation induced by the selected features is obtained in closed form through the frozen decoder, making the procedure end-to-end differentiable. We demonstrate SLI on a Transformer-based transcriptomic foundation model for two clinical prediction tasks, treatment response in inflammatory bowel disease and synovial endotype in rheumatoid arthritis, where curated domain knowledge enables independent validation, and compare it against established attribution methods. On both tasks, SLI recovers compact feature sets that closely reproduce the model’s predictions, recur across patients, and align with domain-relevant structure.

Primary area:
interpretability and explainable AI
Keywords:
Explainability, Mechanistic Interpretability, Sparse Autoencoders, Feature Attribution, Counterfactual Explanations, Latent Feature Selection, Foundation Models for Biology, Transcriptomics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_U2SUOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript presents a mechanistic interpretability framework that identifies, for a frozen model and a single input, a sparse set of SAE features that closely reproduces the model’s prediction. The framework casts this inverse problem as a sparse selection problem: stochastic gates over the entire feature dictionary are jointly optimised under an expected-ℓ0 penalty to preserve the prediction, while the counterfactual representation induced by the selected features is obtained in closed form through the frozen decoder, making the procedure end-to-end differentiable. The authors validate the proposed framework on two clinical prediction tasks using transcriptomic foundation models.

Critical strengths

It is an interesting idea that incorporates counterfactual analysis with SAE decomposition of a polysemantic layer in a deep learning network through an optimisation approach. However, in its current state, the manuscript requires significantly more experimentation before it is ready to meet the standards for publication at the ICLR conference.

Critical weaknesses
  1. The authors need to benchmark their approach against similar works that are not included in the current evaluation, such as: https://openreview.net/forum?id=7X5GcRLHhx (potential plagiarism)

  2. The authors only evaluate one SAE and do not examine additional models. Monosemanticity cannot be verified through a comparison involving only one SAE, particularly without reporting any monosemanticity scores. See the relevant literature: https://neurips.cc/virtual/2025/loc/san-diego/poster/119210. No analysis of alive and dead features in the SAE model is included. Furthermore, no information about the SAE, such as sparsity metrics, is presented, and no hyperparameter tuning of the SAE training is reported.

  3. The attribution techniques may be biased due to inter-method variability. The authors should either include more post-hoc attribution methods or consider using the optimisation approaches proposed in the following works: https://www.sciencedirect.com/science/article/pii/S266665102500004X or https://arxiv.org/html/2601.17952v2

  4. The authors are masking a frozen SAE and optimising the masks until a counterfactual observation occurs. However, there is a potential bias regarding how much the fixed choice of SAE layer (in your case, layer 15 out of 24) contributes to the prediction. Further experimentation is needed to justify the sensitivity of the SAE's layer-level decomposition.

Questions for authors

You are masking a frozen SAE and optimising the masks until a counterfactual observation occurs. However, there is a potential bias regarding how much the fixed choice of SAE layer (in your case, layer 15 out of 24) contributes to the prediction. How did you address this problem?

6 comments
AuthorsOpenReview ↗

Response to Reviewer U2SU (1/2)

We thank the reviewer for the thorough evaluation and helpful pointers to related work and evaluation practices. We have incorporated all suggested references and conducted the requested analyses, strengthening the manuscript in response to the reviewer’s feedback.

1. TimeSAE and the plagiarism concern. We thank the reviewer for pointing us to TimeSAE (Oublal et al., ICML 2026), of which we were not aware when we prepared the submission. The revision cites and discusses it in Section 2 (line 110) and adapts its transferable parts as baselines. We respectfully disagree that the submission plagiarises it, and explain the similarities and differences between the two methods.

  • Method. Despite their differences in contribution and approach, the two methods do share the idea of intervening on a sparse code and decoding it to build a counterfactual, which predates both and was cited in the original submission (Marks et al., 2025). However, SLI's other ingredients, also cited there, differ from TimeSAE's: SLI masks the hidden states of the explained model so that its output is preserved (De Cao et al., 2020), whereas TimeSAE's masks act on its own decoder's terms in input space, and SLI's masks are hard-concrete ℓ0\ell_0 gates (Louizos et al., 2018). The two methods differ in what they explain and how (table below).

  • Baselines from TimeSAE. The two methods answer different questions (which inputs drive a black-box model, against which internal features suffice for one prediction), so a full TimeSAE has no common output or metric with SLI; we compare the parts they share. Its Fx\mathcal{F}_\mathbf{x} removal score is the necessity test we now added (Section 5.2 line 476, A.16). Its Algorithm 1, a continuous, gradient-based change of a pre-selected subset of concepts, is now added as a baseline (Section 5.2 line 472, A.18): adapted to preserving the prediction with an ℓ1\ell_1 penalty in place of the pre-selected subset, it rescales a few latents rather than selecting a set. Read as a set, its top-mm latents (m=∣K∗∣m = |K^\ast|) give ∣Δ∣‾=1.48\overline{|\Delta|} = 1.48 (t001) and 0.240.24 (t014), against 0.0920.092 and 0.1350.135 for SLI's K∗K^\ast. In TimeSAE's own direction, flipping the predicted class, removing the 22 to 44 latents chosen by SLI's gates flips every t001 query. TimeSAE's additive change, applied to the same layer-1515 SAE code with the tail model as the black box and to all latents with an ℓ1\ell_1 penalty that keeps the change small in place of Algorithm 1's pre-selected subset, instead moves a median of 966966 of 2,0482{,}048 latents.

AspectTimeSAESLI
What is explainedThe input-output behaviour of a black-box time-series modelWhich features of an internal layer of a frozen model a prediction depends on
Where the SAE dictionary livesTrained on the input time seriesTrained on token embeddings at layer ℓ\ell, then frozen
Use of the model's outputs in training the SAE dictionaryYes: label-fidelity and contrastive (InfoNCE) losses, plus compositional consistencyNo: the SAE is trained on reconstruction alone; only the gate optimisation uses the model's outputs
What is optimisedEncoder and decoder end to end, amortised over the datasetOnly the gate logits of one input, over a fixed SAE dictionary
Counterfactual targetA different label (ycf≠y\mathbf{y}^{cf} \neq \mathbf{y})The same prediction, with as few features as possible
MasksMasks on the decoder's interaction terms, part of the explainer's decoder; the mask of the highest-order term, which involves all concepts, serves as the input saliency maskHard-concrete gates on the SAE latents, fitted per input against the frozen model, giving a set of features
OutputConcept activations and a saliency mask over time steps and features from the decoderA discrete set of dictionary features selected under an expected-ℓ0\ell_0 penalty
EvaluationAUPRC, AUP and AUR against ground-truth saliency; distributional similarity (KL, MMD, KDE) in and out of distribution; Fx\mathcal{F}_\mathbf{x} removal score; order-faithfulness of causal effectsBinary fidelity and class preservation; size-matched controls; necessity; agreement with attribution methods and Quantus metrics; robustness to gate seeds, SAEs, λz\lambda_z, layers and the <cls> embedding; off-distribution checks; biological validation
AuthorsOpenReview ↗

Response to Reviewer U2SU (2/2)

2. One SAE, monosemanticity, dead latents and SAE tuning.

  • SAE statistics and tuning (A.2, A.11, Table 9). A.2 already reported the SAE's selection and training: five-fold cross-validation over E∈{4,8,16,32}E \in \{4, 8, 16, 32\} and k∈{16,32,64}k \in \{16, 32, 64\} by the splice error (Fig. 4). We now compare different SAEs (A.10) and report their statistics in A.11: the layer-1515 SAEs have no dead latents on held-out or fitting tokens, at most 0.50.5% of latents active on fewer than 0.10.1% of tokens (Fig. 9), and 97.097.0 to 98.098.0% of variance explained. The TopK SAEs activate 6464 latents per token, the ReLU SAE 103103 on t001 and 5252 on t014 on average.

  • More than one SAE (A.10). We now compare the adopted SAE with four more TopK seeds, a ReLU, a JumpReLU and pooled-data SAEs. Fidelity and set size carry over, individual latents do not, and the gene programmes largely do (see our answer to doz6, W1).

  • Monosemanticity (A.11, Fig. 10). We now report the monosemanticity score MS of Pach et al. (NeurIPS 2025, Eq. 9), adapted to genes: the activation-weighted mean Gene Ontology (GO) similarity between the genes of a unit's 5050 top tokens. SAE latents are more selective than layer-1515 neurons: their top tokens cover a median of 33 (t001) and 22 (t014) genes against 55 to 88, and their MS is higher (0.610.61 and 0.970.97 against 0.320.32 to 0.530.53). This mostly reflects gene specificity: over pairs of different genes, the average latent is no more GO-coherent than a neuron or random tokens, and only the latents of Fig. 3 exceed random (2.42.4 and 2.82.8 times). We therefore removed "largely monosemantic" from the introduction (lines 53-54). At the level of concepts, as in InterPLM (Simon & Zou, 2025), SAEs beat neurons: with 256256 units each, five TopK SAEs cover 4.84.8 to 8.18.1 curated gene sets (Hallmark, PanglaoDB, CellMarker) on t001 and 2121 to 3131 on t014, against 11 and 11 to 66 for neurons, which do no better than random directions, and about 00 with genes shuffled. Most SAE latents encode gene identity (7474% and 8787%); the others, context latents that vary with the patient, are more often cell-type enriched, and SLI selects mostly them (7878% and 7474%, against 2323% and 1212% of random latents). On t014, 7070% of its selections are cell-type enriched, against 3030 to 5454% for matched controls; on t001 SLI is level with them.

3. Inter-method variability and optimisation-based alternatives (Section 5.2 line 423, Table 1, A.15). The submission compared SLI with Shapley values and IG. We added single-latent ablation (Heimersheim & Nanda, 2024), Gradient$\times$Activation (Shrikumar et al., 2017), KernelSHAP (Lundberg & Lee, 2017), and Shapley values and IG aligned with SLI's objective: seven rankings (lines 423-426, Table 1). They agree with one another only moderately more than with SLI (mean pairwise top-mm Jaccard 0.520.52 and 0.370.37, against 0.450.45 and 0.270.27 with K∗K^\ast; lines 469-471). Against single-method bias, we implemented the weighted consensus of Mamalakis et al. (2025, Eq. 4; A.15). It agrees with K∗K^\ast about as well as the best single method (AUROC 0.900.90 and 0.880.88), but its top-mm sets are far less sufficient (1.191.19 against 0.0920.092 on t001; 0.430.43 against 0.1350.135 on t014; Table 1), like every method's (1.111.11 to 1.681.68 and 0.410.41 to 0.500.50; lines 432-435): no ranking, alone or combined, yields a sufficient set. In Quantus metrics (Hedström et al., 2023; Table 13), SLI has the lowest complexity (0.460.46 and 0.320.32, others 0.720.72 to 0.900.90), the highest Gini (0.920.92 and 0.940.94) and a faithfulness that is mid-range on t001 and second highest on t014 (0.440.44 and 0.470.47, others 0.220.22 to 0.630.63); as the first two favour sparse outputs, we rely on sufficiency. The trained optimisers of Mamalakis et al. (2025; 2026) output continuous maps, not sets, optimised for explanation-quality metrics; we discuss both in Section 2 (lines 113-116).

4. Layer 15. See doz6, Q4 (A.12). Layer 1515 is a middle layer, where <cls> and the gene embeddings share the prediction. With λz\lambda_z swept at five other layers, earlier layers need more latents for equal fidelity, at layer 2222 gene latents explain little of the prediction, and layer 1919 is more faithful at equal size but leaves most t014 sets empty at its knee.

5. Additional models. We agree that the evaluation covers a single foundation model, and Section 6 (line 524) states this as a limitation. In the discussion phase we prioritised the robustness and baseline analyses requested by all three reviewers: five gate seeds, three SAE architectures, six layers, five attribution and ablation methods (seven rankings with the aligned variants) and their consensus (Table 1), size-matched controls, and a continuous-gate baseline derived from TimeSAE (A.18). We are currently running SLI on a second foundation model, but with the computing resources available we may not have the results before the end of the discussion phase.

I thank the authors for their response. Even if the authors addressed several of my concerns, my remaining concern is the evaluation on a single foundation model. To explain further, even if the ablations and sensitivity analyses demonstrate the robustness of the results across configurations of the given foundation model and case, they do not demonstrate generalizability across a variety of models. Given that the authors proposed a general interpretability framework, such validation is necessary to support its broader applicability and to rule out model-specific effects. Without such evidence, the generalizability of the proposed approach remains insufficiently demonstrated.

For this reason, I maintain my original score.

Reviewer_doz6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Sparse autoencoders (SAEs) decompose hidden representations into interpretable features, but do not reveal which of those features a given prediction depends on. To address this lack, the authors propose Sparse Latent Intervention (SLI), an l0-penalized method, which they describe as model-agnostic, for the selection of a sparse set of latent features based on their ability to reproduce a model’s predictions.

The method combines frozen SAEs with hard-concrete gates, which act as trainable per-input masks over the SAE dictionary, allowing the method to identify the most relevant features for each input. It is validated on a transcriptomic foundation model (EVA-RNA) across two clinical classification tasks (t001: treatment response in inflammatory bowel disease; t014: synovial endotype in rheumatoid arthritis), where fewer than 1% of the dictionary’s latent variables can closely reproduce the model’s predictions.

Critical strengths
  1. The paper clearly formulates the problem and offers a simple yet fitting solution. Hard-concrete gates make the discrete choice of which SAE feature to keep, and the count of kept features, trainable. Decoding the gated code through the frozen SAE decoder keeps the whole procedure end-to-end differentiable. Unlike activation patching, SLI selects a joint feature set, pruning redundant features rather than scoring them separately, while retaining and measuring prediction fidelity.
Critical weaknesses
  1. The evidence is not robust enough to support the claims. The objective is non-convex and optimized stadistically, and the authors acknowledge that the selected set is not unique, yet every explanation comes from a single SAE and a single gate-optimization run.

  2. The paper shows that a small set of features can reproduce the model’s prediction, but this does not necessarily mean the model depends on those features. Since the unselected features are removed, the model receives an input it was not trained on, and the optimizer may simply find features that help recover the original output. It would be useful to test whether removing the selected features from the full representation changes the prediction.

Questions for authors
  1. On t014, 13 of the 76 patients return the empty set, and despite that, the model can reproduce the prediction using only the CLS token. Can you include an ablation where you intervene on the CLS token as well to put this into perspective?

  2. In lines 402–408, you reported the fidelity comparison for t014 but didn’t include the same measurements for t001. Could you include how their top feature sets perform in terms of fidelity?

  3. Could you report the set intersection as λ_z varies? As it’s claimed to “give similar explanations”.

  4. Why did you choose Layer 15? It would be interesting to see the SAE results from different layers.

  5. The gates are shared across all gene tokens, so each latent is either kept or removed for every token in a sample. Would using separate gates for each token change which latents are selected or improve prediction fidelity? If not, why sharing the gates is a reasonable choice?

5 comments
AuthorsOpenReview ↗

Response to Reviewer doz6 (1/2)

We thank the reviewer for the careful assessment and the insightful suggestions, which have helped us strengthen the empirical validation and clarify the scope of our method.

W1. Single SAE and single optimisation run.

  • Gate seeds (A.9). Over five seeds of the gate noise, mean error is ∣Δ∣‾=0.094±0.013\overline{|\Delta|} = 0.094 \pm 0.013 (t001) and 0.132±0.0070.132 \pm 0.007 (t014), and mean set size ∣K∗∣‾=13.3±0.8\overline{|K^\ast|} = 13.3 \pm 0.8 and 6.4±0.36.4 \pm 0.3. Individual sets differ (mean pairwise Jaccard index J=0.57J = 0.57 and 0.600.60), but 1717 of the 1818 t001 latents and all 1818 t014 latents of Fig. 3 are selected under every seed (Table 8). Inspired by stability selection (Meinshausen & Bühlmann, 2010), we report each latent's selection frequency across gate seeds as an empirical measure of selection stability. It tracks importance: dropping from K∗K^\ast a latent chosen by all five seeds raises ∣Δ∣|\Delta| by a median 0.700.70 (t001) and 0.0980.098 (t014), against 0.0950.095 and 0.0220.022 for a latent chosen by one seed.

  • Other SAEs (A.10, Table 7). With four more TopK seeds, mean error is ∣Δ∣‾=0.082\overline{|\Delta|} = 0.082 to 0.1160.116 on t001 and 0.1260.126 to 0.1700.170 on t014, at similar set sizes. A JumpReLU SAE gives the lowest error on t014 among the SAEs trained on the same tokens (0.1100.110; 0.0850.085 on t001, with larger sets). A ReLU SAE has higher errors (0.1400.140 on t001, 0.2160.216 on t014) and leaves more empty sets on t014 (2727 of 7676), which follows from its poorer reconstruction.

  • What carries over and what does not. As reported for language-model SAEs (Paulo & Belrose, 2025), our dictionaries trained with different seeds share few latents: only 11 to 44% of latents have a decoder match above 0.70.7 with the adopted SAE. The Fig. 3 latents are among the most reproducible: 3939 to 5656% (t001) and 5050 to 7272% (t014) have an activation match above 0.70.7 in another seed, against 22% and 77% of all latents. 1010 of the 1818 gene programmes of Fig. 3 on each task are recovered in at least three of five TopK SAEs. Per SAE, SLI recovers 77 to 99 (t001) and 88 to 1111 (t014) programmes, against 1.11.1 to 1.91.9 and 3.23.2 to 4.34.3 for random sets of the same size; no random draw reaches SLI's count. Across SAEs, SLI's sets agree as much as Gradient$\times$Activation sets (matched-latent J=0.26J = 0.26 against 0.270.27 on t001, 0.200.20 against 0.200.20 on t014). We understand that individual latents are dictionary-specific and now read Fig. 3 at the level of programmes (Section 5.1 line 405).

W2. Does the model depend on the selected features, or are they only able to reproduce the output?

  • Necessity (Section 5.2 line 476, A.16). As suggested, removing only K∗K^\ast from the full code changes the prediction far more than removing random sets of the same size (mean absolute output change ∣Δnec∣‾=1.67\overline{|\Delta^{\mathrm{nec}}|} = 1.67 against 0.100.10 on t001; 0.370.37 against 0.0120.012 on t014). It changes it about as much as removing top-activation or attribution sets of the same size (1.581.58 to 1.791.79 and 0.320.32 to 0.360.36), but those sets are not sufficient (∣Δ∣‾=1.1\overline{|\Delta|} = 1.1 to 1.71.7 on t001, 0.40.4 to 0.50.5 on t014), so among the evaluated size-matched sets, K∗K^\ast is the only one that combines low sufficiency error with a substantial removal effect. With controls matched on activation mass, K∗K^\ast is more necessary than top activation on t014 (0.370.37 against 0.190.19), but not on t001. Adding to the objective a term that rewards the output change caused by removing the set gives sets that are more sufficient (∣Δ∣‾=0.070\overline{|\Delta|} = 0.070 against 0.0920.092 on t001; 0.1060.106 against 0.1460.146 on the 2828 non-empty t014 sweep queries) and on average more necessary than count- and mass-matched controls on both tasks. We report this as a variant.

  • Off-distribution inputs (Section 6 line 515, A.17). Every sparse counterfactual, K∗K^\ast and controls alike, is 2.62.6 to 33 times farther from its nearest real token embeddings than real tokens are from theirs. However, four checks indicate that this does not drive the results. (i) SLI's error ∣Δ∣|\Delta| does not grow with this distance (Spearman ρ=−0.43\rho = -0.43, p=0.07p = 0.07 on t001; 0.080.08, p=0.54p = 0.54 on t014). (ii) Outputs on K∗K^\ast's counterfactuals always stay within the range of outputs on real samples. (iii) Under mean ablation K∗K^\ast stays more sufficient than size-matched sets (∣Δ∣‾=0.75\overline{|\Delta|} = 0.75 against 1.491.49 and 1.961.96 on t001; 0.270.27 against 0.520.52 and 0.610.61 on t014). (iv) Under resample ablation, which takes removed latents from real donor samples and brings counterfactuals to 1.31.3 to 1.61.6 times the real distance, gates fitted over 88 donors give sets that stay sufficient on 44 unseen donors (∣Δ∣‾=0.30\overline{|\Delta|} = 0.30 against 0.720.72 for top activation on t001; 0.110.11 against 0.350.35 on t014) and beat top activation on 1818 of 1919 t001 queries and on all 3232 t014 queries. These sets explain the prediction relative to typical samples.

AuthorsOpenReview ↗

Response to Reviewer doz6 (2/2)

Q1. Intervening on <cls> and the 13 empty t014 sets. With the layer-1515 <cls> embedding replaced by its fitting-pool mean (one of three treatments in A.13, Table 12), 99 of the 1313 empty sets become non-empty (6.66.6 latents on average, ∣Δ∣‾=0.17\overline{|\Delta|} = 0.17). Overall error is 0.1000.100 on t001 (0.0990.099 with the embedding kept) and 0.180.18 on t014 (0.130.13), with sets that keep 7676 to 7777% of K∗K^\ast. Encoding the <cls> embedding with the gene SAE and gating it with the gene features makes 1212 of the 1313 non-empty and gives 0.1320.132 on t014, with larger sets (∣K∗∣‾=14.9\overline{|K^\ast|} = 14.9). Section 5.1 (line 332 and line 269) now attributes the empty sets to the kept <cls> embedding and states that explanations are conditional on it.

Q2. Fidelity comparison on t001. Added to Section 5.2 (line 432), Fig. 2d and Table 1. On the 1919 t001 queries, K∗K^\ast gives a mean error ∣Δ∣‾=0.092\overline{|\Delta|} = 0.092, against 1.451.45 for the top-activation set and 1.141.14 for random sets of the same size. K∗K^\ast is lower than the top-activation set for all 1919 patients, and 1717 of 190190 random draws beat it. Top-mm sets of every attribution method (its m=∣K∗∣m = |K^\ast| highest-ranked latents) give 1.111.11 to 1.681.68.

Q3. Set intersection across λz. Table 6 (A.8) reports, for neighbouring values of λz\lambda_z, the Jaccard index JJ, the nesting rate and the share of fully nested sets. Sets are approximately nested: 8585 to 100100% of the smaller set lies in the larger. JJ is lower (0.300.30 to 0.480.48 on t001) because neighbouring sets differ in size by a factor of two to three, which bounds it. Increasing λz\lambda_z step by step and starting each run from the gates found at the previous value gives exactly nested sets for every query, at the same fidelity for a given size. The latents selected at the adopted λz\lambda_z and at both its neighbours are mostly Fig. 3 latents (9090% on t001, 8484% on t014). We replaced "neighbouring values of λz\lambda_z give similar explanations" with these statements (Section 5.1 lines 325-326).

Q4. Why layer 15, and results at other layers. We chose layer 1515 as a middle layer of the 2424. The probe reads the <cls> embedding at the last layer, so <cls> carries no patient-specific information at the input and all of it at the output; we expected an intermediate layer to split the prediction between <cls> and the gene embeddings, so that the gene features carry a substantial part of it. At layer 1515 they do (A.12, Tables 10 and 12): with every latent kept, ∣Δ∣‾=0.205\overline{|\Delta|} = 0.205 on t001 and 0.0240.024 on t014; removing every gene latent raises it to 1.1241.124 and 0.4990.499, and replacing <cls> by its fitting-pool mean, with every latent kept, to 0.6850.685 and 0.3200.320.

To test how sensitive the results are to this choice, we ran SLI at layers 33, 77, 1111, 1919 and 2222, each with its own SAE selected by the same splice-error criterion (A.12, Tables 10 and 11, Fig. 11). With λz\lambda_z kept at its layer-1515 value, set size falls with depth (∣K∗∣‾=103.1\overline{|K^\ast|} = 103.1, 12.612.6 and 1.21.2 at layers 33, 1515 and 2222 on t001; 10.210.2, 6.46.4 and 0.50.5 on t014), and at layer 2222 many sets are empty (88 of 1919 and 6060 of 7676). As set size depends on λz\lambda_z, we also swept λz\lambda_z at every layer, on the 1919 t001 and 3232 t014 queries of the λz\lambda_z sweep, and compared layers at the size of the layer-1515 sets (∣K∗∣‾=12.6\overline{|K^\ast|} = 12.6 and 4.84.8, with ∣Δ∣‾=0.099\overline{|\Delta|} = 0.099 and 0.1410.141). Earlier layers are less faithful: to reach the layer-1515 error, layer 33 needs ∣K∗∣‾=166.9\overline{|K^\ast|} = 166.9 on t001 and 6.86.8 on t014. Later layers are more faithful at that size (layer 1919: ∣Δ∣‾=0.063\overline{|\Delta|} = 0.063 and 0.0500.050; layer 2222: 0.0730.073 and 0.0230.023), partly because removing every gene latent changes the prediction less there. At layer 2222 it changes it three to four times less than at layer 1515 (Table 10), so the selected gene latents explain little of the prediction, and with λz\lambda_z re-tuned per layer at the knee of each error-size curve, most t014 sets there are empty (2929 of 3232).

Layer 1515 is therefore a representative intermediate layer rather than an optimal one: earlier layers need more latents for the same fidelity, at layer 2222 the gene latents explain little of the prediction, and layer 1919 gives slightly more faithful sets than layer 1515 at equal size, but at its knee leaves most t014 sets empty (1818 of 3232). Section 5.1 (line 260) now summarises this comparison.

Q5. Per-token gates. Please see our answer to Reviewer TALM, W3 (A.14): per-token gates give no fidelity gain for the same number of distinct latents, need about 3737 times more parameters, and give scattered (token, latent) pairs that cannot be read as gene programmes (Section 4.2 lines 215-217).

Thank you for your responses and additional experiments.

W1: One thing: "Class specific latents on t014" paragraph should reflect Table 8, as only two of the six lymphoid enriched latents discussed there are recovered in at least three of five TopK SAEs.

W2: Thank you, this addresses my concern.

Reviewer_TALMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Sparse Latent Intervention (SLI), a method for finding a small set of jointly sufficient latent features that explain a model's prediction while preserving it. SLI combines a frozen SAE decoder with jointly optimised hard-concrete gates and an expected-L0 sparsity penalty. Evaluated on two clinical transcriptomic tasks, it produces compact feature sets that approximately preserve predictions and overlap with known biological programmes.

Critical strengths

1/Addresses a meaningful limitation: identifies jointly sufficient latent feature sets, not just individual feature rankings. 2/Mathematically coherent formulation; appropriately acknowledges non-convexity and lack of global-minimum guarantees. 3/Features are biologically plausible, overlapping established programmes and endotype-associated latents.

Critical weaknesses

1/In Section 4.3, the paper states "After convergence, the gates are read out deterministically as; All reported fidelities are binary", the training objective uses stochastic relaxed gates while evaluation thresholds deterministic gates at full activation, so train-to-evaluation discrepancy can change both fidelity and support size; sensitivity to the criterion z* > 0 is not quantified.

2/The paper states "SLI would allow intervention on the <cls> embedding as well, but we keep it", the issue is that explanations are only conditional on the unchanged intermediate <cls> representation, which may already encode substantial predictive information, so results should be framed as conditional rather than exhaustive.

3/ In Section 4.2, it shows "A gate vector z ∈ [0,1]K, shared across token positions, masks it as Az = A diag(z)", one global gate per latent applies across all gene-token positions simultaneously, limiting token-specific attribution and potentially conflating distinct roles of the same latent at different positions.

4/ The paper states "SLI returns a set of features; attribution methods return a ranking", the issue is that there is no direct size-matched fidelity comparison against top-ranked Shapley/Integrated Gradients features under the same binary intervention, and the scalar attribution target vs multi-logit fidelity metric are not aligned.

Questions for authors

see the above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer TALM

We sincerely thank the reviewer for the thoughtful and constructive feedback, and for recognizing the significance and potential of our proposed approach.

W1. Relaxed gates in training, binary gates in evaluation. The discrepancy does not arise in practice. After convergence the hard-concrete gates are exactly 00 or 11, except 22 of 38,91238{,}912 gates on t001 and none of 155,648155{,}648 on t014. We compared five read-outs of the same converged gates (Table 5, A.7): stochastic gates as in training (averaged over 6464 noise samples), the deterministic continuous z∗z^\ast (the gates without noise), and binary masks at z∗>0z^\ast > 0, ≥0.5\geq 0.5 and ≥0.9\geq 0.9. The mean error ranges from 0.0970.097 to 0.1030.103 on t001 and is 0.1340.134 for every read-out on t014; set size changes by at most 0.40.4 latents. Section 4.3 (lines 235-237) now states that all reported errors use binary gates and that the read-out barely changes the error; A.7 states that training uses stochastic gates and evaluation binary ones.

W2. Explanations are conditional on the kept <cls> embedding. We agree and now say so in Section 5.1 (line 269): the explanations name gene features that suffice given the layer-1515 <cls> embedding. To measure how much this embedding carries, we reran SLI under three treatments of it (A.13, Table 12): replacing it by its fitting-pool mean or by the decoded empty code, and encoding it with the gene SAE and gating it with the gene features. With the mean embedding, the gene latents alone still reproduce the prediction: ∣Δ∣‾=0.100\overline{|\Delta|} = 0.100 against 0.0990.099 on t001, and 0.180.18 against 0.130.13 on t014. Sets grow (∣K∗∣‾=14.0\overline{|K^\ast|} = 14.0 and 13.913.9) but keep 7676 to 7777% of K∗K^\ast and every latent of Fig. 3. The decoded empty code (bpre\mathbf{b}_{\mathrm{pre}}) is far from any real <cls> embedding and is a poor replacement: even with every gene latent kept, the error is 5.115.11 on t001 and 0.720.72 on t014 (against 0.2050.205 and 0.0240.024 with the embedding kept), and SLI reaches only ∣Δ∣‾=0.402\overline{|\Delta|} = 0.402 and 0.4880.488, with 55.155.1 and 22.122.1 latents. Gating the <cls> embedding together with the gene features gives errors as low as with it kept (0.0850.085 and 0.1320.132, against 0.0990.099 and 0.1340.134), with larger sets, because the gene SAE must also reconstruct this embedding.

W3. One gate per latent shared across tokens. We implemented one gate per non-zero (token, latent) entry, about 76,50076{,}500 gates per query (A.14, Fig. 12). Plotted against the number of distinct latents, per-token gates trace the same error curve as shared gates on t001 and a worse one on t014 (2020 latents at 0.1160.116, against 6.46.4 latents at 0.1340.134 with shared gates). Each selected latent is kept on only 22 to 77% of the tokens where it is active, so the explanation becomes a list of 99 to 12,40012{,}400 (token, latent) pairs; at matched size its latents overlap little with K∗K^\ast (Jaccard 0.160.16 to 0.240.24). Shared gates may merge position-specific roles of a latent, but in our experiments per-token gating brings no fidelity gain, needs about 3737 times more parameters, and gives scattered (token, latent) pairs that cannot be read as gene programmes; Section 4.2 (lines 215-217) now says so. Token-specific roles can still be read from each selected latent's activations, which SLI leaves unchanged.

W4. Size-matched attribution baselines and the target mismatch. Table 1 (line 442) now reports, for every method, the mean binary error ∣Δ∣‾\overline{|\Delta|} of its top-mm set (the mm latents it ranks highest, with m=∣K∗∣m = |K^\ast| the size of SLI's set): the absolute output error ∣Δ∣|\Delta| (Section 5.1) when only those mm latents are kept under SLI's binary intervention, averaged over queries; Fig. 2d shows the Shapley and IG sets on both tasks. On t001 SLI's K∗K^\ast gives ∣Δ∣‾=0.092\overline{|\Delta|} = 0.092, against 1.111.11 for both Shapley values and IG; on t014 it gives 0.1350.135, against 0.470.47 and 0.450.45. We also added four baselines (A.15): single-latent ablation (Heimersheim & Nanda, 2024), Gradient$\times$Activation (Shrikumar et al., 2017), KernelSHAP (Lundberg & Lee, 2017) and the consensus of Mamalakis et al. (2025, Eq. 4). Their top-mm sets give ∣Δ∣‾=1.51\overline{|\Delta|} = 1.51, 1.681.68, 1.411.41 and 1.191.19 on t001, and 0.420.42, 0.460.46, 0.480.48 and 0.430.43 on t014. To align the target, we recomputed Shapley values and IG for the quantity SLI minimises, the squared error over all logits ("aligned" rows). Agreement with K∗K^\ast does not consistently increase: precision@mm, the share of the top-mm latents that SLI also selects, goes from 0.670.67 to 0.600.60 (Shapley values) and 0.620.62 to 0.590.59 (IG) on t001, and from 0.410.41 to 0.440.44 and 0.380.38 to 0.320.32 on t014. The aligned top-mm sets remain far less sufficient than K∗K^\ast (∣Δ∣‾=1.38\overline{|\Delta|} = 1.38 and 1.351.35 on t001, 0.410.41 and 0.500.50 on t014). The gap therefore comes from scoring latents one at a time rather than selecting them jointly, not from the choice of target.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Concluding Remarks: Reviewer Feedback Has Substantially Strengthened Our Work

Dear Area Chair and Reviewers,

Thank you for taking the time to review our work and for the thoughtful and constructive feedback received so far. We are grateful that the initial reviews recognised the interest and potential of our approach, highlighting that it addresses a meaningful limitation and offers a simple yet fitting solution to the problem of identifying jointly sufficient latent features that closely reproduce the model’s predictions. The reviews also noted the mathematically coherent formulation and the biologically plausible features SLI uncovers, which overlap established gene programmes underlying the clinical predictions.

The Reviewers suggested strengthening the work with further evidence of robustness, broader baselines, a test of whether the model relies on the selected features, and a closer look at the SAE and our design choices. We have conducted extensive additional experiments and analyses and carefully responded to all of their comments and suggestions in detail, point by point.

In particular, we tested robustness across gate seeds, SAE seeds and architectures, network layers, sparsity levels, and treatments of the <cls> embedding, and clarified that our explanations are conditional on the kept <cls> embedding. Across gate seeds and SAE seeds, fidelity and set size carry over, and the gene programmes we report are largely recovered across dictionaries. We compared SLI at matched set size with five attribution and ablation methods, their weighted consensus, attribution scores recomputed for SLI’s own objective, and a continuous-gate baseline adapted from TimeSAE, a method we now also discuss and show to address a different question from ours. At the same set size, no ranking, alone or combined, yields a set as sufficient as SLI’s. We added a necessity test showing that, among the size-matched sets we evaluated, SLI’s is the only one that both reproduces the prediction and substantially changes it when removed. Several checks, including mean and resample ablation, indicate that off-distribution counterfactuals do not drive the results. Finally, we characterised the SAE with dead-latent, variance-explained and monosemanticity analyses, and showed that per-token gates bring no fidelity gain and that the gate read-out barely changes it.

These additions go well beyond clarifications: they provide substantially stronger empirical evidence for our method, a more comprehensive evaluation against alternative approaches, and a much more nuanced understanding of what the selected features capture and how robustly they can be identified. We believe the revised manuscript is significantly improved and gives a considerably more complete account of our contributions, thanks in large part to the Reviewers’ valuable suggestions.

We hope that our detailed responses and the additional evidence address the points raised in the initial reviews and will be reflected in your final assessments. We would be very happy to discuss any remaining questions or outstanding points, and to provide further clarification or analyses if useful before the discussion period concludes.

Thank you again for your time and for helping us improve this work.

AuthorsOpenReview ↗

Concluding Remarks 2: Findings Replicate on a Second Foundation Model (CpGPT)

Dear Area Chair and Reviewers,

Following the latest comments of Reviewers U2SU and doz6, who both noted that the evaluation covered a single foundation model, we have completed the second case study announced in our earlier response and added it to the revised manuscript (abstract, contributions, Section 5.1 lines 406-409, Section 6 and Appendix A.19 with Fig. 15 and Tables 14 and 15).

We applied SLI unchanged to CpGPT (de Lima Camillo et al., 2026), a DNA-methylation foundation model that differs from EVA-RNA in data modality, tokenisation, architecture and task (age regression on 3232 held-out AltumAge profiles). The findings of the first case study hold: on average 7.87.8 latents reproduce the predicted age within 0.530.53 years; size-matched top-activation, random and attribution-ranked sets, including the consensus of all seven rankings, leave errors of 1.781.78 to 13.513.5 years; removing SLI's set changes the prediction 4545 times more than removing random sets; and the results are stable across gate seeds, SAEs and gating of the <cls> embedding. The details are in our response to Reviewer U2SU.

We also addressed the two remaining points of Reviewer doz6, who raised their rating in response to our first reply: Section 5.1 (lines 390-391) now states that only two of the six lymphoid-enriched latents of t014 recur in at least three of five TopK SAEs (Table 8), and Appendix A.13 now discusses the five t014 queries whose sets remain empty with the mean <cls> embedding: their predictions are the closest to the patient-independent output obtained once all patient-specific information is removed, so no feature is worth its cost, and their predicted class is preserved.

Thank you all again for your time and for helping us improve this work. We hope that our detailed responses and substantial new evidence will encourage a favourable final assessment.

AuthorsOpenReview ↗

Final Note to the Area Chair

Dear Area Chair,

We would like to briefly summarise the outcome of the discussion. We have addressed all requests raised by the reviewers, and all three reviewers now recommend acceptance, with Reviewers U2SU and doz6 raising their ratings following the rebuttal.

We thank the Area Chair and all reviewers for their time and constructive feedback, which has substantially improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.