acceptodds
Under review as a conference paper at ICLR 2027

DUNE: Disentangling Polysemantic Neurons Without Losing Predictive Performance

Abstract

Deep neural networks (DNNs) are widely used, but understanding what they actually learn remains difficult. A major obstacle is that individual neurons often respond to multiple unrelated concepts, obscuring the network's internal decision process. Existing approaches, such as sparse autoencoders, can disentangle these mixed signals into more meaningful, “monosemantic” features, but typically introduce approximation errors that may degrade downstream performance. Consequently, these methods exhibit a Pareto frontier between monosemanticity and functional faithfulness. To address this, we introduce _DUNE_ (**d**isentangling **u**nits with **n**o **e**rror), a method that decomposes DNN neurons into more monosemantic subunits while preserving their original function by design. DUNE identifies concept-specific contributions from the preceding layer and re-routes them into separate subunits whose aggregation exactly recovers the original activation. DUNE can be applied directly to pretrained models, without labels or training of an auxiliary model. Across several models, including DINOv2, DUNE achieves state-of-the-art or competitive monosemanticity while exactly preserving functional faithfulness, substantially advancing the existing Pareto frontier between these properties. DUNE operates efficiently and enables fine-grained interventions such as concept-specific steering.

Primary area:
interpretability and explainable AI
Keywords:
Computer Vision, Explainability, Interpretability, Faithfulness, Monosemanticity

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5URpOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DUNE, a post hoc unsupervised method for disentangling polysemantic units into more monosemantic subunits. For each neuron, DUNE clusters the top-k contributions of the previous (l-1) layer’s units and hypothesizes that each cluster corresponds to a concept. The authors create a subunit for each cluster and compute its incoming weights using the margin-based assignment rule. The model's original function is preserved by summing the subunits’ activations back into the original neuron. The method is compared against PCA, semi-NMF, various SAEs, and a transcoder across several models and achieves competitive results.

Critical strengths
  • The empirical baseline is strong: the method was tested across 5+ models against 10+ methods with good results. This could be a nice addition to an XAI toolbox.
  • The method is simple to use and implement which would make it easy for practitioner to adopt it.
Critical weaknesses
  • There is little novelty to the paper. It is a realization of PURE (neurons are no longer virtual, and HDBSCAN replaces k-means) with ARV unit-normalization from Hesse et al., adapted to the unsupervised setting. As far as I can tell, the method is not compared with its two closest competitors, so it is hard to gauge the improvement PURE provides. The HDBSCAN vs k-means results in Table 14, which are closest to the PURE vs DUNE comparison, are mixed.
  • Some parts are misleading or overstated. For faithfulness, the R^2 score is always 100% by design and is what Fig. 1 reports, which artificially inflates the gain over the Pareto front. While Hesse et al. are cited in related work, the ARV unit-normalization link is never mentioned, even though it is explicitly stated as the measurement used in D8.

-> The updates of the authors mostly adress my concerns

Questions for authors
  • It would be great to have a comparison with PURE and Hesse et al. or unsupervised surrogates to really be able to measure the improvement of DUNE. This is partly done with the knn version of DUNE in the appendix, but having these 2 models or surrogates adapted to the task in Fig. 1 would help to easily see DUNE's contribution.
  • In Fig. 1, the y-axis should be changed to explicitly state that it is R^2 and not faithfulness, or use another, less misleading measure. More generally, the perfect R^2 is presented as a key result, but it is always 100% by design, regardless of whether the split between subunits is relevant. As far as I understand, the random model in Table 14 should also have an R^2 of 100%. While this is discussed in the appendix, it would be great to make it clear in the main paper.
  • The link with Hesse et al. ARV should be clearly stated.
4 comments
AuthorsOpenReview ↗

Official comment to Reviewer 5URp (1/2)

Dear Reviewer 5URp, thank you for noting that "the empirical baseline is strong" and that DUNE is "simple to use and implement". We appreciate your request for comparisons with PURE and Hesse et al. and your suggestion to label reconstruction R² more precisely.

W1/Q1. There is little novelty to the paper. It is a realization of PURE (neurons are no longer virtual, and HDBSCAN replaces k-means) with ARV unit-normalization from Hesse et al., adapted to the unsupervised setting.

We agree that the relationship to PURE and Hesse et al.\ should be made clearer to emphasize our novelty. Our method introduces several design choices that address limitations of these approaches. PURE produces ``virtual neurons'' that are not part of the model's forward pass (l. 139 of the initial submission), which means it can not be used in applications that require explicit neurons, such as model steering (Sec. 4.3). Hesse et al. requires supervision, which makes extension to commonly used unsupervised probing datasets and non-classification models challenging, and was evaluated only on a single ImageNet-trained CNN (ll. 142-144 of the initial submission). Our approach provides an explicit, unsupervised decomposition (see Tab. 1). Its main methodological differences extend beyond the choice of clustering algorithm. DUNE removes the unexplained residual neuron retained by Hesse et al., assignes edges by groups rather than using their all-or-nothing assignments, and substitutes their supervised signal with an unsupervised one (see Sec. 3).

Following the reviewer's suggestion, in the revision, we have added direct comparisons with PURE and Hesse et al. (DPCiNN) to Table 2 as well as a discussion of results in Sec. 4.1 (paragraph on quantitative results). We initially excluded these methods because their virtual or supervised decompositions differ from our explicit, unsupervised setting, making a comparison to DUNE and the considered SAEs not entirely fair. Nevertheless, we agree that comparisons on shared evaluation criteria help to establish their relative strengths and limitations. To make Hesse et al. a suitable comparison, we extended their algorithm from a supervised ResNet-50 to an unsupervised DINOv2 ViT, using the official linear classifier head that was trained through supervised ImageNet-1k classification. Their evaluated construction achieves at most an expansion factor of 3, comprising two class subunits and an unexplained residual unit per parent neuron.

Compared with PURE, DUNE achieves a much higher C-Score (54.9454.94 versus 64.8564.85), indicating higher monosemanticity, without relying on the kk parameter deciding how many clusters to build. In addition, DUNE's explicit decomposition supports interventions in the model's forward pass, as demonstrated by our steering experiment (Sec. 4.3). In contrast, PURE's virtual decomposition cannot directly support this application because its virtual units are not part of the forward pass. DUNE achieves similar semantic coherence to Hesse et al. without requiring supervision (65.1765.17 versus 64.8564.85). Furthermore, DUNE outperforms Hesse et al. by 12.09 percentage points on SIM@1 (22.0022.00 versus 34.0934.09). The SIM@1 score of Hesse et al. is decreased by the residual subunits, which score 9.329.32 compared with 28.3328.33 for the class-conditioned subunits.

These comparisons show that DUNE improves on PURE in C-Score and supports direct feature interventions, while achieving comparable or improved semantic coherence to Hesse et al. in an unsupervised setting, without restricting each split to two classes or relying on an unexplained residual neuron.

AuthorsOpenReview ↗

Official comment to Reviewer 5URp (2/2)

W2/Q2. R$^2$ as a misleading measure for Faithfulness and overstating perfect reconstruction by design.

To clarify the axis label of Fig. 1, we changed it to "Reconstruction faithfulness (R$^2$)"; we will update the wording throughout the paper accordingly. Reconstruction faithfulness is an established measure in related work; a non-exhaustive list of prior work confirming this statement includes [C1, C2, C3].

Furthermore, while DUNE achieves perfect R$^2$ by design, advancing the frontier on both monosemanticity and faithfulness is nontrivial, as proven by random disentanglement and the broad selection of our other baselines. To emphasize this point more directly, we have moved the comparison between DUNE and random disentanglement from the appendix to the main table (Tab. 2) and added a discussion in Sec. 4.1, as suggested by the reviewer.

Q3. Link to ARV attribution in Hesse et al.

Thank you for this suggestion. We already acknowledge Hesse et al. as prior work (ll. 141-144 in the initial submission) and made the connection to "ARV unit-normalization" clearer in Section 3.2 of the revision.

We hope the added comparisons and clarifications address your concerns, especially regarding novelty, and are happy to provide further clarifications if requested.

References

[C1] Rajamanoharan et al.: Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders. arXiv:2407.14435 [cs.LG]

[C2] Dunefsky et al., Transcoders Find Interpretable LLM Feature Circuits. In NeurIPS 2024.

[C3] Dhimoïla et al., Cross-Modal Redundancy and the Geometry of Vision–Language Embeddings. In ICLR 2026.

I thank the authors for their detailed explanation, I have updated my score accordingly.

Dear Reviewer 5URp, we thank you once again for your thoughtful feedback and positive assessment of our work.

Reviewer_ksucOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a new method, DUNE, to decompose polysemantic neurons into interpretable components, without affecting downstream model behavior. For each neuron, the authors compute contributions from previous layer neurons and cluster the computed contribution scores. The obtained clusters are used to decompose weights, and thereby decompose polysemantic neurons, into relatively more monosemantic units. The authors demonstrate greater interpretability using top activating data examples, and also show improved steering results with DUNE. They compare their method with multiple SAEs.

Critical strengths

The paper proposes an interesting method of decomposing individual neurons, instead of entire layers, into monosemantic (meaningful) components. The authors include both top activating data examples, as well as steering experiments, to show that their method is both interpretable and causally relevant.

Critical weaknesses
  1. The authors claim existing interpretability tools like SAEs and transcoders are lossy and thereby affect the model's downstream behavior. However, cross-layer transcoders add back the residual error to ensure that there is no downstream effect on model activity. In this case, what is this paper's specific novelty?

  2. Due to clustering that needs to be performed at each individual neuron (and models have 1000s of neurons at a given layer), this paper's method seems computationally expensive. But the authors claim their method is faster than SAEs. How can this be reconciled?

  3. Duplication and potential splitting due to operating on single neurons: individual concepts may appear across multiple neurons, and may even be split across multiple neurons (specific values may be captured by individual neurons). How does the paper's method deal with this challenge?

  4. While the authors demonstrate concept removal using steering, steering a concept's value is arguably harder and more used. Does their method support this kind of steering?

Questions for authors
  1. Why do the authors cluster contribution scores from the previous layer, instead of clustering the activations at the current layer?

  2. Clustering seems to run separately on each neuron: isn't this very expensive for large models with O(1000) neurons?

  3. What about layer nonlinearity? It seems like the authors decompose incoming weights into a neuron, so assume linearity. Do they decompose neurons before the nonlinearity?

  4. Why should clustering of contribution score lead to semantically meaningful units? In other words, why does the proposed method work at all? Can this method be expected to generalize to other models and modalities (say language) too?

I am happy to consider raising my score if the weaknesses and questions are adequately addressed.

4 comments
AuthorsOpenReview ↗

Official comment to Reviewer ksuc (1/2)

Dear Reviewer ksuc, thank you for your questions and for recognizing that DUNE is "both interpretable and causally relevant".

W1. However, cross-layer transcoders add back the residual error to ensure that there is no downstream effect on model activity. In this case, what is this paper's specific novelty?

We thank the reviewer for pointing out the need for comparing to SAEs with residual correction. Indeed, adding the reconstruction error back into the SAE reconstruction restores the original activation, and hence, can be considered faithful. However, this error remains "left unexplained" by the Cross Layer Transcoder [B1], which could reduce the monosemanticity of the resulting combined representation. Instead, DUNE reconstructs the original neuron through its subunits, without a separate error term, which effectively explains the full representation. In the revision, we extend the related work section and the main table (Tab. 2) to include a BatchTopK baseline, which adds the residual error term back to the reconstruction and therefore achieves perfect reconstruction (BatchTopK+R in Tab. 2; details in experimental setup in 4.1). However, as the residual error term remains unexplained, the C-Score drops by 3.83 p.p. and the SIM@1 score drops by 2.82 p.p. Residual correction therefore matches DUNE's reconstruction faithfulness, but at the cost of lower C-Score, SIM@1 and LLM Rank.

W2/Q2. Clustering seems to run separately on each neuron: isn't this very expensive for large models with O(1000) neurons?

We decomposed all 768 neurons of the DINOv2-B/14 layer sequentially on a single A100 GPU needing ∼2\sim2 minutes per neuron (Sec. D.3 of the initial submission). Since each neuron is decomposed independently, the computation can be distributed across GPUs, making layers with 1,000 or more neurons readily feasible (we included this remark in Sec. D.3 of the revision).

Please note that our disentanglement is a one-time cost just as the training of SAEs. The reported time advantages compared to SAEs are at inference time when DUNE is already disentangled and SAEs are already trained. We made this more explicit in the revision (Sec. 4.2 paragraph on computational efficiency).

Q1. Why do the authors cluster contribution scores from the previous layer, instead of clustering the activations at the current layer? Why should clustering of contribution score lead to semantically meaningful units?

Previous-layer contribution vectors describe how a neuron’s response is assembled from its upstream inputs. Each entry measures the contribution through a particular incoming connection, allowing DUNE to identify groups of contributing connections and partition the neuron’s incoming weights into subunits. Clustering current-layer activation vectors can identify groups of similar inputs, but does not directly indicate which incoming connections account for the target neuron’s response or how to partition its weights (remark added to Sec. 3.2 of the revision).

We expect these contribution patterns to reveal semantically useful groups because different concepts activating the same neuron can depend on different upstream circuits, an assumption also used in PURE and Hesse et al. (ll. 233–234 in the initial submission). Such concepts may produce similar activations in the target neuron while differing in which upstream units drive them. Clustering contribution vectors distinguishes examples according to these upstream patterns.

An empirical comparison of these considerations was already included in Appendix B.9 of the initial submission. We compare DUNE with alternatives that cluster raw activations using HDBSCAN or k-means and use the resulting centroids as feature directions. These alternatives achieve higher C-Scores, whereas DUNE achieves better VLM ranks and SIM@1. DUNE also outperforms random splitting across all three metrics (now Tab. 2 of the revision). These results support contribution-based weight partitioning as a way to construct semantically meaningful subunits.

Q3. Does DUNE decompose neurons before the nonlinearity?

Yes, DUNE restructures the neuron's incoming weights, so the resulting subunits appear before the nonlinearity. We sum these subunits and apply the original nonlinearity, recovering the original output exactly: φ(∑cac)=φ(a)\varphi(\sum_c a^c)=\varphi(a). This allows us to edit individual subunits while retaining the neuron's original function. We clarify this order in the revision at the beginning of our method section (Section 3) and use pre-activation notation consistently.

AuthorsOpenReview ↗

Official comment to Reviewer ksuc (2/2)

W3. individual concepts may appear across multiple neurons, and may even be split across multiple neurons

We agree that cross-neuron redundancy is a limitation of DUNE, as acknowledged in Section 5 (l. 466 of the initial submission). We note that obtaining only one feature per concept is also challenging for SAEs, as illustrated by feature splitting and absorption [B2]. We will expand on this discussion to clarify that DUNE is useful for applications in which the semantic coherence of subunits and exact reconstruction are more important than a lack of cross-neuron redundancy. Our steering experiments is one such example, showing better non-target retention at multiple removal rates than the tested SAEs (Section 4.3 of the initial submission). Additionally, our revision shows that DUNE can be sparsified (see W3/Q2 of reviewer MkzH), which could eliminate some redundancy without sacrificing too much faithfulness.

W4. While the authors demonstrate concept removal using steering, steering a concept's value is arguably harder and more used. Does their method support this kind of steering?

While we agree that steering a concept’s value is an interesting direction, we note that concept removal is also a well-established application, studied, for example, in LEACE [B3] and the concept erasure work of Gandikota et al. [B4]. We therefore view our focus on concept removal as an experimental design choice rather than an inherent limitation of DUNE. That said, DUNE supports changing subunit activations with different signs and magnitudes, and we therefore expect steering a concept's value to be possible as well. To provide initial qualitative evidence for this possibility, in Figure 14 of the revision, we include a qualitative example demonstrating how to steer a concept's value by substituting one concept with another using DUNE.

Q4. Can this method be expected to generalize to other models and modalities (say language) too?

Thank you for asking about generalization. In the Appendix B.6 of the initial submission, we evaluate DUNE across several vision models (Tab. 12-13) and also on to two language models (Gemma 3 270M and Qwen3 0.6B; Tab. 18-20). DUNE remains competitive in the monosemanticity metrics at perfect reconstruction on other vision models, confirming the results from the main paper. On the language models, DUNE improves the mean scores on all four interpretability metrics in each evaluated Gemma layer (Tables 18–19) and on three of the four metrics in Qwen (Table 20), compared with the original units. Throughout, it retains 100% reconstruction R$^2$. In the language domain, some SAEs achieve higher semantic scores, but with lower reconstruction quality.

We thank you for the constructive feedback and hope these clarifications support reconsidering your rating. We welcome further discussion of any remaining concerns.

References

[B1] Ameisen et al. “Circuit Tracing: Revealing Computational Graphs in Language Models.” Transformer Circuits, 2025.

[B2] Chanin et al. “A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders.” NeurIPS, 2025.

[B3] Belrose et al. “LEACE: Perfect Linear Concept Erasure in Closed Form.” NeurIPS, 2023.

[B4] Gandikota et al. “Erasing Concepts from Diffusion Models.” ICCV, 2023.

Dear authors, Thank you for your response. I appreciate your additional clarifications and experiments.

From your explanation "Previous-layer contribution vectors describe how a neuron’s response is assembled from its upstream inputs", it seems to me like DUNE and SAEs decompose activations in different ways. Transcoders may be a more fair comparison, because they also look across model layers and aim to incorporate model computation into their decomposition. A mention of this may be useful to readers of your paper, since SAEs decompose activations without knowledge of upstream (or downstream) circuits.

There are limitations I still see: I don't see a significant increase in monosemanticity-faithfulness performance with DUNE compared to other methods (Table 2), I'm not entirely sure about the generalization of DUNE to other modalities and models (authors acknowledge "In the language domain, some SAEs achieve higher semantic scores, but with lower reconstruction quality"), and believe redundant decomposition--multiple units capturing the same concept--may be an important issue in this method.

However, I believe decomposing neurons using upstream weights is an interesting method for the community and the paper is well-executed. A clear discussion of the above limitations will be useful to anyone who reads the paper.

I will hence raise my score to 3 and recommend acceptance.

Thank you for engaging in discussion during the rebuttals!

Dear Reviewer ksuc, we thank you once again for your thoughtful feedback and positive assessment of our work. We will incorporate the additional discussion into the manuscript.

Reviewer_MkzHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript introduces DUNE, a method for disentangling polysemantic units in neural networks into more monosemantic subunits. DUNE collects top-k activation instances for a target unit, computes contribution vectors from the previous layer, clusters these vectors with HDBSCAN to identify concept-specific groups, and restructures the incoming weights so that each subunit is primarily connected to a concept-specific group of upstream units. Because the subunit weights form an additive partition of the original weights, the sum of subunit activations exactly reconstructs the original unit activation, yielding a lossless, explicit, unsupervised, and stable disentanglement.

Critical strengths
  • Clear and well-motivated problem formulation. The paper correctly identifies the tradeoff between monosemanticity and faithfulness in existing disentanglement methods and proposes a construction that targets both simultaneously.
  • The additive weight partition guarantees that subunit activations sum exactly to the original unit activation. This is a genuine advantage over SAE- and transcoder-based approaches, which introduce reconstruction error.
  • Unlike virtual decompositions such as DNRwCV and PURE, DUNE produces real subunits that can be inspected and modified directly, enabling steering and visualization without post-hoc approximation.
Critical weaknesses
  • Missing citation and positioning relative to architecture-level disentanglement. The manuscript does not cite or discuss Architecture Disentanglement for Deep Neural Networks (Hu et al., ICCV 2021, NAD). NAD also studies disentanglement of deep networks, but at the level of sub-architectures and task-specific information flow, rather than at the level of individual polysemantic units.
  • Per-neuron decomposition creates cross-neuron redundancy. The same concept may appear in subunits of multiple original neurons. The paper mentions correlation-based pruning as a possible remedy, but pruning would sacrifice losslessness. How to control redundancy while preserving exact reconstruction remains unresolved.
  • Dense, signed activations rather than sparse non-negative codes. Unlike SAEs, DUNE does not produce sparse or non-negative features. This may limit interpretability workflows that rely on sparse feature activation, feature monitoring, or concept-set analysis. The steering results are encouraging, but the representational form differs substantially from the SAE literature.
Questions for authors
  1. Qualify monosemanticity claims. State explicitly that the reported metrics are proxies and that DUNE does not guarantee human-interpretable concept alignment. Avoid unqualified “state-of-the-art monosemanticity” language; specify the metrics and settings under which the claim holds.
  2. Discuss the absence of sparsity more directly. Explain how dense, signed subunit activations affect interpretability workflows compared with SAE features. If possible, report a sparse variant or a post-hoc feature-selection procedure and its effect on faithfulness.
2 comments
AuthorsOpenReview ↗

Official comment to Reviewer MkzH

Dear Reviewer MkzH, thank you for highlighting our exact reconstruction which yields "a genuine advantage over SAE- and transcoder-based approaches", and for your suggestions on related work and evaluation.

W1. Missing citation and positioning relative to architecture-level disentanglement.

Thank you for pointing out this complementary work. We added NAD [A1] to the related work section of the revision to clarify the distinction between NAD's task-specific sub-architectures and DUNE's task-agnostic decomposition of individual neurons.

W2. Per-neuron decomposition creates cross-neuron redundancy.

We agree that cross-neuron redundancy is a limitation of DUNE, as acknowledged in Section 5 (l. 466 of the initial submission). We note that obtaining only one feature per concept is also challenging for SAEs, as illustrated by feature splitting and absorption [A2]. We will expand on this discussion to clarify that DUNE is useful for applications in which the semantic coherence of subunits and exact reconstruction are more important than a lack of cross-neuron redundancy. Our steering experiments is one such example, showing better non-target retention at multiple removal rates than the tested SAEs (Section 4.3 of the initial submission). Additionally, our revision shows that DUNE can be sparsified (see W3/Q2), which could eliminate some redundancy without sacrificing too much faithfulness.

W3/Q2. If possible, report a sparse variant or a post-hoc feature-selection procedure and its effect on faithfulness.

Thank you for the suggestion, in response we added an additional ablation on sparsity (Sec. 4.2), where we compare DUNE and three standard SAEs (Vanilla, TopK, and BatchTopK) at varying sparsity levels alongside a more detailed discussion. For DUNE we can use the same disentanglement at different sparsity levels (Sec. D4), the SAEs have to be retrained for each level.

In the high-sparsity regime (less than 1% of active subunits), TopK and BatchTopK outperform sparse DUNE in R$^2$ and C-Score. As more subunits are kept, DUNE's C-Score coherence improves, while TopK and BatchTopK lose coherence with more active subunits. For example, at 4.2% to 12.5%, DUNE achieves higher C-Score than TopK and BatchTopK SAE, and almost equal as Vanilla SAE (53.553.5 vs. 53.353.3). However, TopK and BatchTopK retain higher reconstruction R$^2$ at these budgets. In low sparsity regimes, DUNE continues to improve coherence while outperforming the SAEs in R$^2$ reconstruction.

Q1. State explicitly that the reported metrics are proxies and that DUNE does not guarantee human-interpretable concept alignment.

We agree that monosemanticity metrics are proxies and that high scores do not guarantee alignment with human-interpretable concepts. We discuss this limitation in Section 5 (l. 472 of the initial submission), but this should be made clearer in our main claims. In the revision, we updated the abstract, and the introduction to clarify that "state-of-the-art or competitive" refers to the reported metrics and experimental settings.

We hope these clarifications address your concerns, and we welcome further questions.

References

[A1] Hu et al. “Architecture Disentanglement for Deep Neural Networks.” ICCV, 2021.

[A2] Chanin et al. “A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders.” NeurIPS, 2025.

Dear Reviewer MkzH, thank you once again for your thoughtful feedback and positive assessment of our work. We hope that our response has adequately addressed your remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.