- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The manuscript presents a mechanistic interpretability framework that identifies, for a frozen model and a single input, a sparse set of SAE features that closely reproduces the model’s prediction. The framework casts this inverse problem as a sparse selection problem: stochastic gates over the entire feature dictionary are jointly optimised under an expected-ℓ0 penalty to preserve the prediction, while the counterfactual representation induced by the selected features is obtained in closed form through the frozen decoder, making the procedure end-to-end differentiable. The authors validate the proposed framework on two clinical prediction tasks using transcriptomic foundation models.
Critical strengths
It is an interesting idea that incorporates counterfactual analysis with SAE decomposition of a polysemantic layer in a deep learning network through an optimisation approach. However, in its current state, the manuscript requires significantly more experimentation before it is ready to meet the standards for publication at the ICLR conference.
Critical weaknesses
-
The authors need to benchmark their approach against similar works that are not included in the current evaluation, such as: https://openreview.net/forum?id=7X5GcRLHhx (potential plagiarism)
-
The authors only evaluate one SAE and do not examine additional models. Monosemanticity cannot be verified through a comparison involving only one SAE, particularly without reporting any monosemanticity scores. See the relevant literature: https://neurips.cc/virtual/2025/loc/san-diego/poster/119210. No analysis of alive and dead features in the SAE model is included. Furthermore, no information about the SAE, such as sparsity metrics, is presented, and no hyperparameter tuning of the SAE training is reported.
-
The attribution techniques may be biased due to inter-method variability. The authors should either include more post-hoc attribution methods or consider using the optimisation approaches proposed in the following works: https://www.sciencedirect.com/science/article/pii/S266665102500004X or https://arxiv.org/html/2601.17952v2
-
The authors are masking a frozen SAE and optimising the masks until a counterfactual observation occurs. However, there is a potential bias regarding how much the fixed choice of SAE layer (in your case, layer 15 out of 24) contributes to the prediction. Further experimentation is needed to justify the sensitivity of the SAE's layer-level decomposition.
Questions for authors
You are masking a frozen SAE and optimising the masks until a counterfactual observation occurs. However, there is a potential bias regarding how much the fixed choice of SAE layer (in your case, layer 15 out of 24) contributes to the prediction. How did you address this problem?
6 comments
Response to Reviewer U2SU (2/2)
2. One SAE, monosemanticity, dead latents and SAE tuning.
-
SAE statistics and tuning (A.2, A.11, Table 9). A.2 already reported the SAE's selection and training: five-fold cross-validation over and by the splice error (Fig. 4). We now compare different SAEs (A.10) and report their statistics in A.11: the layer- SAEs have no dead latents on held-out or fitting tokens, at most % of latents active on fewer than % of tokens (Fig. 9), and to % of variance explained. The TopK SAEs activate latents per token, the ReLU SAE on t001 and on t014 on average.
-
More than one SAE (A.10). We now compare the adopted SAE with four more TopK seeds, a ReLU, a JumpReLU and pooled-data SAEs. Fidelity and set size carry over, individual latents do not, and the gene programmes largely do (see our answer to doz6, W1).
-
Monosemanticity (A.11, Fig. 10). We now report the monosemanticity score MS of Pach et al. (NeurIPS 2025, Eq. 9), adapted to genes: the activation-weighted mean Gene Ontology (GO) similarity between the genes of a unit's top tokens. SAE latents are more selective than layer- neurons: their top tokens cover a median of (t001) and (t014) genes against to , and their MS is higher ( and against to ). This mostly reflects gene specificity: over pairs of different genes, the average latent is no more GO-coherent than a neuron or random tokens, and only the latents of Fig. 3 exceed random ( and times). We therefore removed "largely monosemantic" from the introduction (lines 53-54). At the level of concepts, as in InterPLM (Simon & Zou, 2025), SAEs beat neurons: with units each, five TopK SAEs cover to curated gene sets (Hallmark, PanglaoDB, CellMarker) on t001 and to on t014, against and to for neurons, which do no better than random directions, and about with genes shuffled. Most SAE latents encode gene identity (% and %); the others, context latents that vary with the patient, are more often cell-type enriched, and SLI selects mostly them (% and %, against % and % of random latents). On t014, % of its selections are cell-type enriched, against to % for matched controls; on t001 SLI is level with them.
3. Inter-method variability and optimisation-based alternatives (Section 5.2 line 423, Table 1, A.15). The submission compared SLI with Shapley values and IG. We added single-latent ablation (Heimersheim & Nanda, 2024), Gradient$\times$Activation (Shrikumar et al., 2017), KernelSHAP (Lundberg & Lee, 2017), and Shapley values and IG aligned with SLI's objective: seven rankings (lines 423-426, Table 1). They agree with one another only moderately more than with SLI (mean pairwise top- Jaccard and , against and with ; lines 469-471). Against single-method bias, we implemented the weighted consensus of Mamalakis et al. (2025, Eq. 4; A.15). It agrees with about as well as the best single method (AUROC and ), but its top- sets are far less sufficient ( against on t001; against on t014; Table 1), like every method's ( to and to ; lines 432-435): no ranking, alone or combined, yields a sufficient set. In Quantus metrics (Hedström et al., 2023; Table 13), SLI has the lowest complexity ( and , others to ), the highest Gini ( and ) and a faithfulness that is mid-range on t001 and second highest on t014 ( and , others to ); as the first two favour sparse outputs, we rely on sufficiency. The trained optimisers of Mamalakis et al. (2025; 2026) output continuous maps, not sets, optimised for explanation-quality metrics; we discuss both in Section 2 (lines 113-116).
4. Layer 15. See doz6, Q4 (A.12). Layer is a middle layer, where <cls> and the gene embeddings share the prediction. With swept at five other layers, earlier layers need more latents for equal fidelity, at layer gene latents explain little of the prediction, and layer is more faithful at equal size but leaves most t014 sets empty at its knee.
5. Additional models. We agree that the evaluation covers a single foundation model, and Section 6 (line 524) states this as a limitation. In the discussion phase we prioritised the robustness and baseline analyses requested by all three reviewers: five gate seeds, three SAE architectures, six layers, five attribution and ablation methods (seven rankings with the aligned variants) and their consensus (Table 1), size-matched controls, and a continuous-gate baseline derived from TimeSAE (A.18). We are currently running SLI on a second foundation model, but with the computing resources available we may not have the results before the end of the discussion phase.
I thank the authors for their response. Even if the authors addressed several of my concerns, my remaining concern is the evaluation on a single foundation model. To explain further, even if the ablations and sensitivity analyses demonstrate the robustness of the results across configurations of the given foundation model and case, they do not demonstrate generalizability across a variety of models. Given that the authors proposed a general interpretability framework, such validation is necessary to support its broader applicability and to rule out model-specific effects. Without such evidence, the generalizability of the proposed approach remains insufficiently demonstrated.
For this reason, I maintain my original score.
Response to Reviewer U2SU (1/2)
We thank the reviewer for the thorough evaluation and helpful pointers to related work and evaluation practices. We have incorporated all suggested references and conducted the requested analyses, strengthening the manuscript in response to the reviewer’s feedback.
1. TimeSAE and the plagiarism concern. We thank the reviewer for pointing us to TimeSAE (Oublal et al., ICML 2026), of which we were not aware when we prepared the submission. The revision cites and discusses it in Section 2 (line 110) and adapts its transferable parts as baselines. We respectfully disagree that the submission plagiarises it, and explain the similarities and differences between the two methods.
Method. Despite their differences in contribution and approach, the two methods do share the idea of intervening on a sparse code and decoding it to build a counterfactual, which predates both and was cited in the original submission (Marks et al., 2025). However, SLI's other ingredients, also cited there, differ from TimeSAE's: SLI masks the hidden states of the explained model so that its output is preserved (De Cao et al., 2020), whereas TimeSAE's masks act on its own decoder's terms in input space, and SLI's masks are hard-concrete ℓ0 gates (Louizos et al., 2018). The two methods differ in what they explain and how (table below).
Baselines from TimeSAE. The two methods answer different questions (which inputs drive a black-box model, against which internal features suffice for one prediction), so a full TimeSAE has no common output or metric with SLI; we compare the parts they share. Its Fx removal score is the necessity test we now added (Section 5.2 line 476, A.16). Its Algorithm 1, a continuous, gradient-based change of a pre-selected subset of concepts, is now added as a baseline (Section 5.2 line 472, A.18): adapted to preserving the prediction with an ℓ1 penalty in place of the pre-selected subset, it rescales a few latents rather than selecting a set. Read as a set, its top-m latents (m=∣K∗∣) give ∣Δ∣=1.48 (t001) and 0.24 (t014), against 0.092 and 0.135 for SLI's K∗. In TimeSAE's own direction, flipping the predicted class, removing the 2 to 4 latents chosen by SLI's gates flips every t001 query. TimeSAE's additive change, applied to the same layer-15 SAE code with the tail model as the black box and to all latents with an ℓ1 penalty that keeps the change small in place of Algorithm 1's pre-selected subset, instead moves a median of 966 of 2,048 latents.