acceptodds
Under review as a conference paper at ICLR 2027

Sparse AutoEncoders Reveal That MoE Experts Exhibit Human-Interpretable Fine-Grained Specialization

Abstract

Mixture-of-Experts (MoE) models route each token to a few of many experts, yet what an individual expert is responsible for remains unsettled. Existing analyses count how often an expert is selected on domain-labeled corpora, an instrument that fixes the taxonomy in advance, stays coarse, and records selection rather than contribution. We instead study specialization in feature space by training a sparse autoencoder (SAE) on the output of an MoE layer. Since this output is a router-weighted sum of expert outputs and the SAE encoder is affine, the pre-activation of every latent decomposes exactly over the experts. Aggregated over a corpus, this gives each expert a dominant set of latents whose activation it principally drives. On layers 4, 30 and 46 of Qwen3-30B-A3B, contribution is far more concentrated than gating, most latents have only one or two dominators, and the overlap between dominant sets concentrates on a few expert pairs. Auto-interpreting these sets reveals a division of labor below coarse domains, with distinct experts for Java and front-end web code, or for geometry and for arithmetic and polynomials, while the coarse domains themselves take very unequal numbers of experts.

Primary area:
interpretability and explainable AI
Keywords:
SAE, LLM, interpretability, MoE

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_31L7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the interpretability of Mixture-of-Experts (MoE) models by applying a Sparse Autoencoder (SAE) to the outputs of MoE layers. The authors find that expert contributions are more concentrated in SAE features and provide empirical results to support this observation. Furthermore, they use LLMs to summarize the inputs that activate certain features and analyze their meanings.

Critical strengths
  1. MoE models have shown impressive performance in recent years. However, the understanding of the division of labor among experts is still quite limited. Consequently, understanding the interpretability of MoE models is an important problem.
  2. Combining SAEs with MoE models is a natural idea. It has been verified that sparsity can significantly improve the monosemanticity of dense LLM models. As a result, it is valuable to evaluate whether the sparsity in MoE models can further improve the interpretability of LLMs
  3. Some empirical results are interesting and insightful in this paper. For example, the authors show that the contributions of experts are more concentrated with a sparse autoencoder, and different experts share limited overlapping contributions
Critical weaknesses
  1. My main concerns lie in whether the analysis in this paper can be applied in practice. For example, it would be interesting if the features can be used to provide guidance for the training process of MoE models.
  2. SAE-based interpretability has been studied extensively in dense models. I wonder whether MoE models have any advantages over dense models in terms of interpretability. A fair comparison with dense models using SAEs would be useful to better understand the role of expert specialization.
  3. Although the examples on selected experts in Section 5 are interesting, I think more quantitative results are needed. For example, there exist quantitative evaluation methods in SAEBench. It would be much better if the authors can provide more empirical verifications based on that.
  4. The experiments are mainly conducted on a specific model with a fixed size. As the training process and architectures are significantly different of different MoE models. I think it would be interesting to verify the results on more models.
Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to reviewer 31L7

We sincerely thank the reviewer for their constructive feedback and recognition of our key strengths: (1) understanding the interpretability of MoE models is an important problem, (2) combining SAEs with MoE models is a natural idea, and (3) Our empirical results are interesting and insightful in this paper. We also appreciate the reviewer's valuable suggestions for improving our paper. We address your concerns below:

w1: whether the analysis in this paper can be applied in practice.

Thank you for your question. We would like to clarify that our work focuses more on developing a more accurate and fine-grained recipe to understanding specialization in MoE models, rather than on their applications. We hope our work can help researchers address the understudied question of how specialization organized within MoE models. Potential applications include leveraging insights into specialization to understand how model parameters allocate knowledge, as well as to guide strategies such as data mixing ratios during training. Thank you for your suggestion, and we will add discussions on this topic in the Discussion section.

w2: whether MoE models have any advantages over dense models in terms of interpretability.

Thank you for your question. We would like to clarify that we do NOT claim MoE models are more interpretable than dense models. Rather, we use SAE as an interpretability tool to investigate MoE models, aiming to understand and discover how experts specialize at the fine-grained feature level. We agree with you that comparison of the interpretability between MoE and dense model architectures is an interesting research direction. However, such a comparison is beyond the scope of our current work, and we leave it for future investigation.

w3: more quantitative results are needed. For example, there exist quantitative evaluation methods in SAEBench.

Thank you for your suggestion. While SAEBench is an evaluation method for SAE, our work focuses on interpreting Experts. However, this is an excellent idea for verifying whether our interpretations of the experts are accurate using an approach similar to that in SAEBench. Following your recommendation, we adopted an approach inspired by the Automated Interpretability evaluation in SAEBench to quantitatively assess the accuracy of our expert interpretations.Specifically, for each expert, we collected 20 positive examples that were top-1 routed to that expert and 20 negative examples that were not routed to that expert (i.e., not among its top-8) from the validation set. We then provided GPT-5.5 with only the expert's interpretation and these samples, and asked it to predict which samples would activate the expert. We evaluated the predictions against the actual routing assignments and obtained an average accuracy of 0.782 ± 0.137 across experts (mean ± standard deviation), compared with random guessing's accuracy of 0.5. These results provide quantitative evidence that our expert interpretations are consistent with the actual routing behavior. We appreciate the reviewer’s suggestion, which helped strengthen our evaluation.

w4: verifying the results on more models are need

Thank you for your suggestion. We would like to clarify that the primary contribution of our work is a new methodology, a more accurate and fine-grained framework for understanding expert specialization in MoE models, rather than an attempt to establish universal specialization patterns across different models. We use Qwen3-30B-A3B as our research subject and a concrete case study to demonstrate the effectiveness of our methodology and analyze the specialization patterns observed in this model. We acknowledge that expert specialization can depend heavily on model architecture and training recipes, and therefore do not claim that the specific patterns we identify necessarily generalize to all MoE models. Due to the substantial computational costs, we were unable to extend our SAE experiments to a broader range of models or larger-scale models. Nevertheless, this limitation does not restrict the applicability of our methodology. The proposed framework can also be applied to other MoE models. To facilitate further validation, we will open-source our datasets, models, and training code, enabling the research community to reproduce our experiments and investigate expert specialization in other MoE models.

Thank you again for your question and suggestions to improve our work!

Reviewer_4m3AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes an SAE-based framework for analysing fine-grained expert specialisation in MoE models. By exploiting the linearity of the SAE encoder, the authors decompose the MoE layer output into contributions from individual experts to learned features. Experiments on Qwen3-30B-A3B reveal concentrated expert contributions, limited overlap between experts' dominant feature sets, and interpretable fine-grained specialisation patterns.

Critical strengths
  1. A well-motivated and meaningful analytical framework. I appreciate the paper's approach to expert specialisation from two complementary perspectives. First, it moves beyond predefined, coarse-grained domain labels by using SAE-learned features, enabling a more fine-grained analysis of expert behaviour. Second, rather than relying solely on routing frequency, it examines how individual expert outputs contribute to different learned features. I find the combination particularly well motivated. The authors exploit a natural mathematical relationship between the MoE output and the SAE encoder to develop a more informative perspective on expert specialisation.

  2. Interesting empirical observations supporting the proposed framework. The paper provides meaningful observations about expert contribution patterns across different layers and a large number of inputs. In particular, the concentration of expert–feature associations after aggregation across inputs is interesting, since concentrated contributions for individual inputs do not necessarily imply consistent associations with specific experts across the dataset.The fine-grained semantic examples further illustrate the potential of the proposed analysis.

Critical weaknesses

The observations do not yet provide a sufficiently clear and unified understanding of fine-grained expert specialisation.

While I appreciate the proposed approach and find the empirical observations reasonable and interesting, I did not find a sufficiently clear and convincing overarching conclusion about how MoE experts organise their fine-grained specialisation.

The paper reports several observations, including contribution concentration, expert–feature associations, and interpretable semantic preferences. However, I would have liked to see these observations further consolidated into a more explicit and meaningful mathematical or statistical relationship, supported by corresponding empirical evidence.

I do not think this necessarily requires a new theorem or rigorous theoretical proof. Rather, I was hoping for a clearer statistical regularity or unified explanation that connects these observations and provides a deeper understanding of fine-grained expert specialisation.

Questions for authors

see the weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer 4m3A

We sincerely thank the reviewer for their constructive feedback and recongnition of our key strengths: (1) using fine-grained SAE-learned features instead of coarse, predefined domain labels, (2) measuring expert contributions by decomposing the SAE encoder rather than simply counting expert selections, and (3) identifying concentrated expert–feature associations by aggregating across inputs. We also appreciate the reviewer's valuable suggestions for improving our paper.

Our central finding is that MoE experts exhibit a hierarchical organisation of largely distinct but partially overlapping specialisations. We agree with the reviewer's concern that our observations do not yet provide a sufficiently clear and unified understanding of fine-grained expert specialisation. We clarify this organisational structure through the following observations:

How the experts specialization organized. From experts' explanations we grouped these explanations into coarse domains (e.g. Computing). (a) Every domain is covered by several experts (Sec. 5.3), and each expert mainly takes a finer-grained subfield (or theme in other word) inside its domain (Sec. 5.2). (b) These subfields are not a hard partition. The dominant sets behind them are largely disjoint, but they share at their boundaries (Secs. 4.2 and 4.3). (c) Expert pairs inside the same domain can share a substantial number of features, and the strongly overlapping examples are neighbouring subfields, such as two experts on JavaScript web programming inside Computing (Sec. 5.3). Expert pairs across domains also commonly share features, but only a few per pair (see the supplementary analysis below).

Supplementary Figure. To provide a more unified statistical view of this organisation, we conduct an additional analysis of pairwise Jaccard similarities between experts' dominant feature sets. Specifically, we reorganise the middle panel of Figure 8 (which reports pairwise Jaccard similarities) by grouping experts according to the broad domains identified in Section 5.3, placing experts from the same domain adjacent to one another along both axes. The resulting heatmap is presented below. The white boxes delineate the domain groups, the diagonal is masked, and the colour scale is saturated at (J=0.05) to make the overlap patterns more visible.

please refer the link

The resulting pattern provides direct empirical support for the above observations: large Jaccard similarities are concentrated predominantly within the diagonal domain blocks, whereas similarities across different domain blocks are sparse and generally weak. For example, the expert pair with the largest overlap, E83–E105, consists of two experts from the Text Format domain. In contrast, most pairs of experts belonging to different domains have little or no overlap in their dominant feature sets.

Supplementary analysis. We computed the same-domain share over all expert pairs, grouped by overlap. Let Djj′=1D_{jj'}=1 if experts jj and j′j' are in the same coarse domain and Djj′=0D_{jj'}=0 otherwise, and let Jjj′J_{jj'} be the Jaccard similarity of their dominant sets. For the 8,128 unordered pairs at layer 30, the estimated Pr⁡(Djj′=1∣Jjj′)\Pr(D_{jj'}=1\mid J_{jj'}) is:

JJPairsSame domain
003,2905.5%
(0,10−4](0, 10^{-4}]7697.8%
(10−4,10−3](10^{-4}, 10^{-3}]2,22813.1%
(10−3,10−2](10^{-3}, 10^{-2}]1,54423.4%
(10−2,0.05](10^{-2}, 0.05]27744.4%
>0.05>0.052095.0%

The result shows that as the Jaccard similarity increases, the probability that the experts selected in such pairs belong to the same coarse domain rises. This indicates from another perspective that overlaps mainly occur within the same coarse domain.

Revision. We sincerely thank the reviewer for this suggestion that we should provide a sufficiently clear and unified understanding of fine-grained expert specialisation. We will add the aforementioned content in the revision and describe how expert specialisation is organised. We find that the lack of a clear narrative thread in our original text has led to the reviewer’s confusion. We will restructure Section 5, and move the discussion of feature cloud versus token cloud in Section 5.1, which is tangential to our main narrative, to the appendix to better highlight our core arguments.

Thank you again for suggestions to improve our work!

Reviewer_BonROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes using SAEs (sparse autoencoders) to study fine-grained specialization of experts in MoEs. The main idea is to decompose the SAE latent activations into expert contributions and use these contributions to characterize the expert specialization. However, I found the paper difficult to follow as many sentences are dense and combine multiple ideas. This made it difficult for me to form a clear understanding of the method, and thus fairly evaluate the novelty and/or relevance of the work.

Critical strengths

I am unable to fairly evaluate the strengths due to the lack of clarity in the presentation.

Critical weaknesses

The main weakness is the clarity and presentation. For example,

In the abstract:

  • "Existing analyses count how often an expert is selected on domain-labeled corpora, an instrument that fixes the taxonomy in advance, stays coarse, and records selection rather than contribution."
    • "instrument", is unusual and unclear here.
    • what "fixes the taxonomy", and what "stays coarse"? Is it domain labels?
    • "selection rather than contribution", how is contribution defined?
  • "Aggregated over a corpus, this gives each expert a dominant set of latents whose activation it principally drives. On layers 4, 30 and 46 of Qwen3-30B-A3B, contribution is far more concentrated than gating, most latents have only one or two dominators, and the overlap between dominant sets concentrates on a few expert pairs."
    • "Aggregated over a corpus", what is being aggregated?
    • "dominant set of latents", "dominators", how are they defined?
    • "Contribution" is compared with "gating", without defining these quantities.

Similarly in the introduction, the sentences are convoluted:

  • "SAE latents approximate the features the model itself uses, and are discovered from its activations rather than stipulated in advance."
  • "In resolution, we replace corpus-level selection frequency with a per-token, per-expert contribution score, which both avoids attributing every token of a document to a single domain and measures functional contribution rather than mere co-occurrence."
Questions for authors

I don't have specific questions about the method as I was unable to understand it sufficiently.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updates of our revision

Dear Reviewers,

Thank you very much for your time, thoughtful reading, and constructive feedback. Based on your suggestions, we have uploaded a revised manuscript with clearer writing, a better-organized narrative, and additional analyses. All newly added or revised text in the manuscript is highlighted in blue, and all previously reported experimental results remain completely unchanged.

Our main revisions focus on two goals: simplifying complex sentences to make the paper easier to read, and reorganizing the narrative to highlight our core story. Specifically, we made the following changes:

  1. Clarifying our methodological contribution (Abstract, Section 1, and Section 6). We now explicitly frame our approach as a general feature-space framework for analyzing expert specialization in any MoE layer, rather than a study limited to a single model. We then present Qwen3-30B-A3B as a concrete case study of this framework.
  2. Improving readability throughout the paper. We rewrote long and complex sentences across all sections into shorter, clearer statements.
  3. Restructuring Section 4 around hypothesis-driven checks. We reorganized our structural analysis into three clear steps:
    • Section 4.1: We first verify that our contribution score captures genuine feature preferences beyond router gating weights alone.
    • Section 4.2 (Feature perspective): We test whether individual features have dedicated experts. Most latents (85.1%–92.7%) are dominated by only one or two experts, supporting specialization from the feature side.
    • Section 4.3 (Expert perspective): We check both expert coverage and pairwise overlap. All 128 experts hold nonempty dominant sets without collapsing onto a few hubs, and different experts hold nearly disjoint sets. Together, Sections 4.2 and 4.3 show that the model is specialized in feature space. We also moved the degree plots (Figure 3) and full Jaccard heatmaps (Figure 4) from the appendix into Section 4, and moved the detailed summary table (Table 3) to Appendix C.
  4. Reorganizing Section 5 around how experts divide labor. We moved the comparison between explanation word clouds and token word clouds to Appendix D, since it is tangential to our main storyline. Section 5 now follows a clear progression from coarse domains to fine subfields and boundary overlap:
    • Section 5.1: Interpreting individual experts through their dominant sets.
    • Section 5.2: Recovering 11 coarse domains at layer 30 and showing that expert allocation across domains is strongly uneven.
    • Section 5.3: Showing that within each coarse domain, experts further specialize in distinct fine-grained subfields.
    • Section 5.4: Showing that feature overlap between dominant sets occurs mainly between neighboring subfields within the same coarse domain.
  5. Unifying the summary in Section 6 (Conclusion). We added a structured summary of this hierarchical division of labor at layer 30 of Qwen3-30B-A3B and reiterated our methodological and empirical contributions.

In addition, we added two complementary analyses in Section 5.4 to examine how dominant-set overlap relates to coarse domains at layer 30:

  • Domain-grouped Jaccard heatmap (Figure 7). We reordered the 128 experts of layer 30 along both axes of the pairwise Jaccard matrix so that experts from the same coarse domain (Section 5.2) sit adjacent to one another. White boxes mark the 11 diagonal domain blocks, the main diagonal is masked, and the color scale saturates at J=0.05J = 0.05. The resulting heatmap shows a clear block-diagonal pattern. High-similarity entries fall predominantly inside the diagonal domain blocks, whereas entries across different domains are sparse and weak. Across all 8,128 expert pairs, the mean Jaccard similarity within the same domain is 5.2×10−35.2 \times 10^{-3}, roughly five times the cross-domain mean of 1.0×10−31.0 \times 10^{-3}.
  • Same-domain share across Jaccard bins (Table 2). To quantify this visual pattern, we grouped all 8,128 unordered expert pairs at layer 30 into six bins of Jaccard similarity JJ and computed the fraction of pairs belonging to the same coarse domain in each bin. Across all pairs, the unconditional same-domain rate is 12.8%. As JJ increases, the same-domain share rises monotonically from 5.5% at J=0J = 0 and 7.8% for J∈(0,10−4]J \in (0, 10^{-4}], to 23.4% for J∈(10−3,10−2]J \in (10^{-3}, 10^{-2}], 44.4% for J∈(10−2,0.05]J \in (10^{-2}, 0.05], and 95.0% (19 out of 20 pairs) for J>0.05J > 0.05. Without binning, a randomly chosen same-domain pair has a larger JJ than a randomly chosen cross-domain pair with probability 0.712. Both results confirm that meaningful feature overlap occurs primarily between neighboring subfields within the same coarse domain.

Thank you once again for your insightful comments and suggestions, which have greatly helped us improve our paper!

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.