- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the interpretability of Mixture-of-Experts (MoE) models by applying a Sparse Autoencoder (SAE) to the outputs of MoE layers. The authors find that expert contributions are more concentrated in SAE features and provide empirical results to support this observation. Furthermore, they use LLMs to summarize the inputs that activate certain features and analyze their meanings.
Critical strengths
- MoE models have shown impressive performance in recent years. However, the understanding of the division of labor among experts is still quite limited. Consequently, understanding the interpretability of MoE models is an important problem.
- Combining SAEs with MoE models is a natural idea. It has been verified that sparsity can significantly improve the monosemanticity of dense LLM models. As a result, it is valuable to evaluate whether the sparsity in MoE models can further improve the interpretability of LLMs
- Some empirical results are interesting and insightful in this paper. For example, the authors show that the contributions of experts are more concentrated with a sparse autoencoder, and different experts share limited overlapping contributions
Critical weaknesses
- My main concerns lie in whether the analysis in this paper can be applied in practice. For example, it would be interesting if the features can be used to provide guidance for the training process of MoE models.
- SAE-based interpretability has been studied extensively in dense models. I wonder whether MoE models have any advantages over dense models in terms of interpretability. A fair comparison with dense models using SAEs would be useful to better understand the role of expert specialization.
- Although the examples on selected experts in Section 5 are interesting, I think more quantitative results are needed. For example, there exist quantitative evaluation methods in SAEBench. It would be much better if the authors can provide more empirical verifications based on that.
- The experiments are mainly conducted on a specific model with a fixed size. As the training process and architectures are significantly different of different MoE models. I think it would be interesting to verify the results on more models.
Questions for authors
See Weaknesses.
Response to reviewer 31L7
We sincerely thank the reviewer for their constructive feedback and recognition of our key strengths: (1) understanding the interpretability of MoE models is an important problem, (2) combining SAEs with MoE models is a natural idea, and (3) Our empirical results are interesting and insightful in this paper. We also appreciate the reviewer's valuable suggestions for improving our paper. We address your concerns below:
w1: whether the analysis in this paper can be applied in practice.
Thank you for your question. We would like to clarify that our work focuses more on developing a more accurate and fine-grained recipe to understanding specialization in MoE models, rather than on their applications. We hope our work can help researchers address the understudied question of how specialization organized within MoE models. Potential applications include leveraging insights into specialization to understand how model parameters allocate knowledge, as well as to guide strategies such as data mixing ratios during training. Thank you for your suggestion, and we will add discussions on this topic in the Discussion section.
w2: whether MoE models have any advantages over dense models in terms of interpretability.
Thank you for your question. We would like to clarify that we do NOT claim MoE models are more interpretable than dense models. Rather, we use SAE as an interpretability tool to investigate MoE models, aiming to understand and discover how experts specialize at the fine-grained feature level. We agree with you that comparison of the interpretability between MoE and dense model architectures is an interesting research direction. However, such a comparison is beyond the scope of our current work, and we leave it for future investigation.
w3: more quantitative results are needed. For example, there exist quantitative evaluation methods in SAEBench.
Thank you for your suggestion. While SAEBench is an evaluation method for SAE, our work focuses on interpreting Experts. However, this is an excellent idea for verifying whether our interpretations of the experts are accurate using an approach similar to that in SAEBench. Following your recommendation, we adopted an approach inspired by the Automated Interpretability evaluation in SAEBench to quantitatively assess the accuracy of our expert interpretations.Specifically, for each expert, we collected 20 positive examples that were top-1 routed to that expert and 20 negative examples that were not routed to that expert (i.e., not among its top-8) from the validation set. We then provided GPT-5.5 with only the expert's interpretation and these samples, and asked it to predict which samples would activate the expert. We evaluated the predictions against the actual routing assignments and obtained an average accuracy of 0.782 ± 0.137 across experts (mean ± standard deviation), compared with random guessing's accuracy of 0.5. These results provide quantitative evidence that our expert interpretations are consistent with the actual routing behavior. We appreciate the reviewer’s suggestion, which helped strengthen our evaluation.
w4: verifying the results on more models are need
Thank you for your suggestion. We would like to clarify that the primary contribution of our work is a new methodology, a more accurate and fine-grained framework for understanding expert specialization in MoE models, rather than an attempt to establish universal specialization patterns across different models. We use Qwen3-30B-A3B as our research subject and a concrete case study to demonstrate the effectiveness of our methodology and analyze the specialization patterns observed in this model. We acknowledge that expert specialization can depend heavily on model architecture and training recipes, and therefore do not claim that the specific patterns we identify necessarily generalize to all MoE models. Due to the substantial computational costs, we were unable to extend our SAE experiments to a broader range of models or larger-scale models. Nevertheless, this limitation does not restrict the applicability of our methodology. The proposed framework can also be applied to other MoE models. To facilitate further validation, we will open-source our datasets, models, and training code, enabling the research community to reproduce our experiments and investigate expert specialization in other MoE models.
Thank you again for your question and suggestions to improve our work!