- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that experts in multimodal MoE models specialize by domain and introduces ExpertLens, which identifies such experts without domain data. For adaptation, the selected experts are fine-tuned while the others are frozen, speeding up adaptation. On Qwen3-VL-30B-A3B and Kimi-VL-A3B, the resulting models perform on par with full fine-tuning (FFT) and are better on some datasets.
Critical strengths
A practical recipe for adapting an MoE to a specialized domain by training only a subset of experts, with reasonable empirical support on two backbones. On Qwen3-VL-30B-A3B, it matches or slightly exceeds FFT while updating 21.7% to 47.0% of the parameters and training about 4.0× faster. On Kimi-VL-A3B, it also matches FFT (mean 48.0 vs. 47.4) and trains faster (1.53×).
Critical weaknesses
- The central premise is that experts in multimodal MoEs specialize by domain, but the domain analysis in Section 3.2 relies on hidden states, and the authors only give a qualitative argument that the structure of the hidden-state distribution could induce domain-specific routing. The paper does not include a direct analysis linking the two, for example, whether tokens in the same hidden-state cluster activate the same experts. Fig. 5A gives only indirect evidence, showing that the experts selected by ExpertLens prefer their domains for medical and remote sensing. However, for math, Figs. 2 and 5A together point to a mismatch: the image-patch hidden states are well separated, yet the math experts show a less clear preference.
- Figs. 2 and 3 show that image activations are more dataset-specific than text activations, and Fig. 1 shows that many experts favor image tokens, yet ExpertLens assigns domains from text alone. The identified experts may therefore not match the target domain, and experts that specialize only on images may go unidentified.
Questions for authors
- Which MedMNIST subsets, and in what proportions, were used in the Section 3 analyses?
- Which modules does the LoRA baseline adapt?
- In Fig. 5B, the text tokens associated with the medical expert E121 are all genetics-related (gene, DNA, genome, ...), whereas its top-activating images are histopathology slides. These are two different subfields of medicine. How should this inconsistency be interpreted?
6 comments
Official Response to Reviewer 3bgx continued
the domain analysis in Section 3.2 relies on hidden states, and the authors only give a qualitative argument that the structure of the hidden-state distribution could induce domain-specific routing. [...] for math, Figs. 2 and 5A together point to a mismatch: the image-patch hidden states are well separated, yet the math experts show a less clear preference.?
We thank the reviewer for this point. You correctly point out that §3.2 examines hidden states, and make an interesting observation about Figure 5A and 2 potentially drawing different conclusions. Below we clarify how the activation clustering links to routing, echoing the main paper (L210-213, 298-300). Afterwards, we clarify why Fig 2 and 5A are not actually a mismatch. Rather, they reflect a subtle difference between whether a domain is specialized upon, and the number of experts that will specialize in it!
In an MoE layer, routing is a direct function of the hidden state. Each expert's router logit is the inner product between its router weight and the token's hidden state, and the top-K logits select the experts. Thus, tokens with nearby hidden states therefore receive similar router logits and tend to select similar experts (L298-300). Figure 2 makes this visible by projecting router weights together with the activations, showing how router weights concentrate near domain clusters of image activations. These analyses provide the intuition that there must exist domain specific experts. Instead of relying on the hidden states of pre-selected domain-specific data to source such specialized experts, ExpertLens enables a data-free manner of expert domain discovery.
With this background, we can clarify why we believe that Figure 2 and Figure 5A are consistent. The narrower specialization of remote sensing and medical domain experts over math ones likely reflects training data statistics. Math data is typically a large portion of VLM pre-training mixtures (e.g. 7% of Cambrian1 - Tong et. al [3], 20.1% of LLaVA-One-Vision - Li et. al [4] and a dedicated STEM diagram understanding subset in Qwen-3 - Qwen3 Team [5], sec 3.2.8), while the other two domains are not. A plausible consequence is that models devote more experts to math, while fewer and narrower experts emerge for rarer domains. This is reflected in ExpertLens outcomes. As our method does not fix the number of experts per domain, it selects 2,768 math experts, compared with 1,103 for medical and 1,290 for remote sensing, which are less represented in pre-training data mixtures. In addition, EL decoding for math involves numbers, symbols, etc., which can be more general than the math domain, so the math experts are more general (that also explains their larger number).
Because of their larger presence and coverage, math experts are less spiky in frequency for their domain, as shown in Fig 5A. The tightness of domain clusters in Fig 2 indicates that activations from the same domains are likely to call similar experts. The clusters are a similar size because the study involves balancing the number of image patches from each domain. But the number of experts that specialize in that domain, and the degree of specialization each one will take, are likely heavily affected by this training data mix bias! Hence what we see in Fig 5A.
We lastly note that this does not limit ExpertLens in practice. On math, our method matches or exceeds full fine-tuning and outperforms LoRA (Fig. 6).
Which modules does the LoRA baseline adapt?
Adapters are applied to the attention projections (q, k, v, o) of every decoder layer and to the routed-expert MLP weights, following Wei et al., 2026 and Liu et al., 2026 (L105). We use rank 32 and α = 64 (rank 64 was also evaluated; rank 32 performed better and is the configuration reported) following Shuttleworth et al. 2025 (L373). These details have been updated in our manuscript at §A.5.1.
Thank you again for taking the time to closely review our work, and the clarification requests to make our results more comprehensive! We put our references below:
[1] Neo, Clement, et al. "Towards interpreting visual information processing in vision-language models." International Conference on Learning Representations. Vol. 2025. 2025.
[2] Coudray, Nicolas, et al. "Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning." Nature medicine 24.10 (2018): 1559-1567.
[3] Tong, Shengbang, et al. "Cambrian-1: A fully open, vision-centric exploration of multimodal llms." Advances in Neural Information Processing Systems 37 (2024): 87310-87356.
[4] Li, Bo, et al. "Llava-onevision: Easy visual task transfer." arXiv preprint arXiv:2408.03326 (2024).
[5] Bai, Shuai, et al. "Qwen3-vl technical report." arXiv preprint arXiv:2511.21631 (2025).
Thank you again for taking the time to review our paper. As the discussion period deadline is approaching, we wanted to briefly follow-up to check if you had any remaining questions or comments. Thank you.
Official Response to Reviewer 3bgx
We thank the reviewer for their review. We appreciate the clarification questions, which we address in our response below. Our manuscript has also been updated to reflect added clarifications and any experiments. Please let us know if you have further questions.
We thank the reviewer for raising this point! We would like to clarify a few different points to show that ExpertLens does not assign domains from text alone. In addition to the explanations below, we will go through the manuscript and update to reflect any potentially misleading sections.
At its core, ExpertLens is a data-free method for classifying experts into their preferred domains by decoding their router weights (§3.4, Eq. 1). Each expert's router weight is a vector in the model's shared latent space, the same space that text-token and image-patch activations occupy. An expert is selected when this vector aligns with a token's representation, whatever the token's modality. ExpertLens reads out the semantics of this vector by projecting it through the LM head (L300-305). The vocabulary serves only as a human-interpretable representation for the latent space; it does not restrict the method to experts that process text.
Further, we would like to note that logit lens (the technique ExpertLens builds upon) has been shown to work equally well on decoding image patch activations as well as text token activations in previous work. Specifically, Neo et. al [1] saw that image patches from a patterned sweater would decode to semantic concepts like “pattern” and “design”, and a plain white patch from the background would decode into “background” while the same white patch on a light switch would decode to “light”. The fact that logit lens is able to sufficiently read out such high level semantics from image patches shows that the LM head is able to take the router weights of experts that preferentially fire for images and decode out their preferred domains.
ExpertLens works well on decoding the preferred domains of both image- and text-aligned experts because during VLM training, activations from both domains become semantically aligned. For example, an expert that fires almost exclusively on patches from histopathology images has a router weight aligned with histopathology-like representations, and decoding that weight yields medical tokens (Figure 5B). Our figures show this directly across both modalities. In Figure 5A, ExpertLens cohorts are evaluated on both image patches and text tokens (L316-319), and we see strong alignment on image patches (this holds across layers, shown in Figure 9 in the Appendix). In Figure 5B, For the medical (E121), math (E76) and remote-sensing (E93) experts shown, the decoded tokens match the domain of the images that most strongly activate each expert.
ExpertLens therefore does not assign domains from text. It does not assign them from images either: it uses no inputs of any modality and identifies domain experts from the model weights alone.
We do not view this as an inconsistency for two reasons. First, ExpertLens operates at the granularity of the target domain. In our experiments the target is medical, and both genetics and histopathology fall within it, so E121 is correctly identified as a medical expert. The decoded tokens show which medical concepts the expert's router weight aligns with most strongly and do not need to name the exact subfield of the images that recruit it. Second, to the best of our understanding, pathology and biopsy imagery are routinely used to study the genetic basis of disease (e.g. Coudray et al. (Nature Medicine, 2018) [2]). At the cohort level, Figure 5A and Figure 9 confirm that ExpertLens-selected medical experts as a group are recruited most strongly by medical datasets.
We specifically use PathMNIST, which are colorectal histopathology patches from 9 distinct tissue classes. For the analysis behind Fig.2, we use 1500 image patches and 1500 text tokens across multiple samples. For the analysis behind Fig.3, we use 2000 tokens each from PathVQA and PathMNIST. We use the tissue class labels as QA answers. Information about specific datasets selected for each analysis can be found in each experiment’s appendix section (Sec A.2.1., A.2.2., A.2.3.) - we will update them to include this subcategory selection information.