acceptodds
Under review as a conference paper at ICLR 2027

Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation

Abstract

Mixture-of-Experts (MoE) architectures scale model capacity through sparse computation, routing each token through only a small subset of experts. In this work, we explore whether this sparsity gives rise to emergent intrinsic organization in multimodal MoEs. We find that experts develop strong semantic specialization across modalities and domains despite not being explicitly trained for modularity. Building on this structure, we introduce ExpertLens, a data-free method that identifies domain-specialized experts directly from pretrained model weights by decoding router weights into semantically meaningful vocabulary tokens. We leverage this specialization for efficient multimodal adaptation by selectively fine-tuning experts relevant to a target domain. Across math, medical, and remote sensing tasks, ExpertLens matches or surpasses full fine-tuning while updating only – of model parameters and achieving a average training speedup, and outperforms LoRA in both adaptation performance and training efficiency. These results show that sparsity introduced for efficiency can give rise to semantic modularity that is directly useful for efficient adaptation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture of experts, adaptation, vlms

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3bgxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that experts in multimodal MoE models specialize by domain and introduces ExpertLens, which identifies such experts without domain data. For adaptation, the selected experts are fine-tuned while the others are frozen, speeding up adaptation. On Qwen3-VL-30B-A3B and Kimi-VL-A3B, the resulting models perform on par with full fine-tuning (FFT) and are better on some datasets.

Critical strengths

A practical recipe for adapting an MoE to a specialized domain by training only a subset of experts, with reasonable empirical support on two backbones. On Qwen3-VL-30B-A3B, it matches or slightly exceeds FFT while updating 21.7% to 47.0% of the parameters and training about 4.0× faster. On Kimi-VL-A3B, it also matches FFT (mean 48.0 vs. 47.4) and trains faster (1.53×).

Critical weaknesses
  1. The central premise is that experts in multimodal MoEs specialize by domain, but the domain analysis in Section 3.2 relies on hidden states, and the authors only give a qualitative argument that the structure of the hidden-state distribution could induce domain-specific routing. The paper does not include a direct analysis linking the two, for example, whether tokens in the same hidden-state cluster activate the same experts. Fig. 5A gives only indirect evidence, showing that the experts selected by ExpertLens prefer their domains for medical and remote sensing. However, for math, Figs. 2 and 5A together point to a mismatch: the image-patch hidden states are well separated, yet the math experts show a less clear preference.
  2. Figs. 2 and 3 show that image activations are more dataset-specific than text activations, and Fig. 1 shows that many experts favor image tokens, yet ExpertLens assigns domains from text alone. The identified experts may therefore not match the target domain, and experts that specialize only on images may go unidentified.
Questions for authors
  1. Which MedMNIST subsets, and in what proportions, were used in the Section 3 analyses?
  2. Which modules does the LoRA baseline adapt?
  3. In Fig. 5B, the text tokens associated with the medical expert E121 are all genetics-related (gene, DNA, genome, ...), whereas its top-activating images are histopathology slides. These are two different subfields of medicine. How should this inconsistency be interpreted?
6 comments
AuthorsOpenReview ↗

Official Response to Reviewer 3bgx

We thank the reviewer for their review. We appreciate the clarification questions, which we address in our response below. Our manuscript has also been updated to reflect added clarifications and any experiments. Please let us know if you have further questions.

Figs. 2 and 3 show that image activations are more dataset-specific than text activations, and Fig. 1 shows that many experts favor image tokens, yet ExpertLens assigns domains from text alone. The identified experts may therefore not match the target domain…

We thank the reviewer for raising this point! We would like to clarify a few different points to show that ExpertLens does not assign domains from text alone. In addition to the explanations below, we will go through the manuscript and update to reflect any potentially misleading sections.

At its core, ExpertLens is a data-free method for classifying experts into their preferred domains by decoding their router weights (§3.4, Eq. 1). Each expert's router weight is a vector in the model's shared latent space, the same space that text-token and image-patch activations occupy. An expert is selected when this vector aligns with a token's representation, whatever the token's modality. ExpertLens reads out the semantics of this vector by projecting it through the LM head (L300-305). The vocabulary serves only as a human-interpretable representation for the latent space; it does not restrict the method to experts that process text.

Further, we would like to note that logit lens (the technique ExpertLens builds upon) has been shown to work equally well on decoding image patch activations as well as text token activations in previous work. Specifically, Neo et. al [1] saw that image patches from a patterned sweater would decode to semantic concepts like “pattern” and “design”, and a plain white patch from the background would decode into “background” while the same white patch on a light switch would decode to “light”. The fact that logit lens is able to sufficiently read out such high level semantics from image patches shows that the LM head is able to take the router weights of experts that preferentially fire for images and decode out their preferred domains.

ExpertLens works well on decoding the preferred domains of both image- and text-aligned experts because during VLM training, activations from both domains become semantically aligned. For example, an expert that fires almost exclusively on patches from histopathology images has a router weight aligned with histopathology-like representations, and decoding that weight yields medical tokens (Figure 5B). Our figures show this directly across both modalities. In Figure 5A, ExpertLens cohorts are evaluated on both image patches and text tokens (L316-319), and we see strong alignment on image patches (this holds across layers, shown in Figure 9 in the Appendix). In Figure 5B, For the medical (E121), math (E76) and remote-sensing (E93) experts shown, the decoded tokens match the domain of the images that most strongly activate each expert.

ExpertLens therefore does not assign domains from text. It does not assign them from images either: it uses no inputs of any modality and identifies domain experts from the model weights alone.

In Fig. 5B, the text tokens associated with the medical expert E121 are all genetics-related (gene, DNA, genome, ...), whereas its top-activating images are histopathology slides. How should this inconsistency be interpreted?

We do not view this as an inconsistency for two reasons. First, ExpertLens operates at the granularity of the target domain. In our experiments the target is medical, and both genetics and histopathology fall within it, so E121 is correctly identified as a medical expert. The decoded tokens show which medical concepts the expert's router weight aligns with most strongly and do not need to name the exact subfield of the images that recruit it. Second, to the best of our understanding, pathology and biopsy imagery are routinely used to study the genetic basis of disease (e.g. Coudray et al. (Nature Medicine, 2018) [2]). At the cohort level, Figure 5A and Figure 9 confirm that ExpertLens-selected medical experts as a group are recruited most strongly by medical datasets.

Which MedMNIST subsets, and in what proportions, were used in the Section 3 analyses?

We specifically use PathMNIST, which are colorectal histopathology patches from 9 distinct tissue classes. For the analysis behind Fig.2, we use 1500 image patches and 1500 text tokens across multiple samples. For the analysis behind Fig.3, we use 2000 tokens each from PathVQA and PathMNIST. We use the tissue class labels as QA answers. Information about specific datasets selected for each analysis can be found in each experiment’s appendix section (Sec A.2.1., A.2.2., A.2.3.) - we will update them to include this subcategory selection information.

AuthorsOpenReview ↗

Official Response to Reviewer 3bgx continued

the domain analysis in Section 3.2 relies on hidden states, and the authors only give a qualitative argument that the structure of the hidden-state distribution could induce domain-specific routing. [...] for math, Figs. 2 and 5A together point to a mismatch: the image-patch hidden states are well separated, yet the math experts show a less clear preference.?

We thank the reviewer for this point. You correctly point out that §3.2 examines hidden states, and make an interesting observation about Figure 5A and 2 potentially drawing different conclusions. Below we clarify how the activation clustering links to routing, echoing the main paper (L210-213, 298-300). Afterwards, we clarify why Fig 2 and 5A are not actually a mismatch. Rather, they reflect a subtle difference between whether a domain is specialized upon, and the number of experts that will specialize in it!

In an MoE layer, routing is a direct function of the hidden state. Each expert's router logit is the inner product between its router weight and the token's hidden state, and the top-K logits select the experts. Thus, tokens with nearby hidden states therefore receive similar router logits and tend to select similar experts (L298-300). Figure 2 makes this visible by projecting router weights together with the activations, showing how router weights concentrate near domain clusters of image activations. These analyses provide the intuition that there must exist domain specific experts. Instead of relying on the hidden states of pre-selected domain-specific data to source such specialized experts, ExpertLens enables a data-free manner of expert domain discovery.

With this background, we can clarify why we believe that Figure 2 and Figure 5A are consistent. The narrower specialization of remote sensing and medical domain experts over math ones likely reflects training data statistics. Math data is typically a large portion of VLM pre-training mixtures (e.g. 7% of Cambrian1 - Tong et. al [3], 20.1% of LLaVA-One-Vision - Li et. al [4] and a dedicated STEM diagram understanding subset in Qwen-3 - Qwen3 Team [5], sec 3.2.8), while the other two domains are not. A plausible consequence is that models devote more experts to math, while fewer and narrower experts emerge for rarer domains. This is reflected in ExpertLens outcomes. As our method does not fix the number of experts per domain, it selects 2,768 math experts, compared with 1,103 for medical and 1,290 for remote sensing, which are less represented in pre-training data mixtures. In addition, EL decoding for math involves numbers, symbols, etc., which can be more general than the math domain, so the math experts are more general (that also explains their larger number).

Because of their larger presence and coverage, math experts are less spiky in frequency for their domain, as shown in Fig 5A. The tightness of domain clusters in Fig 2 indicates that activations from the same domains are likely to call similar experts. The clusters are a similar size because the study involves balancing the number of image patches from each domain. But the number of experts that specialize in that domain, and the degree of specialization each one will take, are likely heavily affected by this training data mix bias! Hence what we see in Fig 5A.

We lastly note that this does not limit ExpertLens in practice. On math, our method matches or exceeds full fine-tuning and outperforms LoRA (Fig. 6).

Which modules does the LoRA baseline adapt?

Adapters are applied to the attention projections (q, k, v, o) of every decoder layer and to the routed-expert MLP weights, following Wei et al., 2026 and Liu et al., 2026 (L105). We use rank 32 and α = 64 (rank 64 was also evaluated; rank 32 performed better and is the configuration reported) following Shuttleworth et al. 2025 (L373). These details have been updated in our manuscript at §A.5.1.

Thank you again for taking the time to closely review our work, and the clarification requests to make our results more comprehensive! We put our references below:

[1] Neo, Clement, et al. "Towards interpreting visual information processing in vision-language models." International Conference on Learning Representations. Vol. 2025. 2025.
[2] Coudray, Nicolas, et al. "Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning." Nature medicine 24.10 (2018): 1559-1567.
[3] Tong, Shengbang, et al. "Cambrian-1: A fully open, vision-centric exploration of multimodal llms." Advances in Neural Information Processing Systems 37 (2024): 87310-87356.
[4] Li, Bo, et al. "Llava-onevision: Easy visual task transfer." arXiv preprint arXiv:2408.03326 (2024).
[5] Bai, Shuai, et al. "Qwen3-vl technical report." arXiv preprint arXiv:2511.21631 (2025).

Thank you again for taking the time to review our paper. As the discussion period deadline is approaching, we wanted to briefly follow-up to check if you had any remaining questions or comments. Thank you.

Reviewer_NjBpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents ExpertLense -- a data-free method allowing to detect the domain specialised experts. After this step, the authors investigate whether having these expert <-> domain dependencies can be used for additional fine-tuning on relevant domains. By enabling additional training stage, the authors manage to update only ~21-47% of model parameters while roughly matching the full fine-tuning accuracy and outperforming LoRA training setup.

The paper flows naturally: first, authors study whether the experts have any specialisation by checking whether textual or visual tokens are routed differently. After that, they check whether a semantic domain-level specialisation exists. The authors extend the approach to a data-free method: instead of using domain-specific examples, ExpertLense uses logit-lens decoding -- mapping router weights through its LM head to predict a distribution over vocabulary tokens. By then prompting QWEN3-VL-30B-A3B to classify the top-10 predicted tokens, the method maps them to domains. Finally, after identifying domain experts, the authors tune the corresponding MLPs and shared decoder parameters on 4k samples, resulting in training ~21-47% model parameters.

The authors compare the partially fine-tuned model against full fine-tuning and LoRA with rk=32. The results show that this training setting can match full fine-tuning and outperform LoRA training. Generally, the method provides an interesting view on domain-specific fine-tuning and provides a lot of interesting insights and analysis.

Critical strengths
  1. The paper presents a very clear idea starting from a general motivation supporting the modality preference in VLM MoE experts and narrowing down to domain specialisation. Building on these insights, the paper proposes the partial fine-tuning method allowing to improve accuracy and being on par with full fine-tuning. The idea and method are very clear, the paper provides rich insights and visualisations. Such methods are usually easy to use and are beneficial for the community.
Critical weaknesses
  1. I think that the comparisons could be further expanded. Regarding the LoRA setup: the resulting model updates ~21-47% of model parameters, and LoRA training uses ranks 32 and 64. While I cannot directly estimate the percentage of updated parameters with these ranks, I assume it would be lower than what ExpertLense updates. As a result, the comparison is between a solid training setup for a part of the model weights vs updating (all?) experts but "a little bit" through a low LoRA rank training.

  2. There is an overlap between training and testing datasets (while respecting train/test splits). While it is acceptable, I find the current approach of model performance reporting a bit confusing: the tables should individually cover the datasets which were not present in training data. And this is possible: e.g., for Math, the training samples were sourced from MMK12 (A5.1), while test datasets cover MMK12, MATHVISTA, and MATHVISION. Also see Q2.

I hesitate between the scores 2 and 3. I like the proposed method, but I believe that the current state of experiments and evaluation does not fully support the claims. Namely, experiments with full training and random experts subset are insightful, but LoRA trainings have ranks which might be too low to match the amount of updated parameters (rk=32,64) and I am curious if domain-specific experts are popular [see Q1]. Secondly, I would like to see the individual per-dataset accuracies [Q2]. I am happy to engage in the further discussions and revisit my score based on the authors' answers to these weaknesses and questions below.

Questions for authors
  1. I think it would be beneficial to additionally check how updating top - ~20-40% most frequently chosen experts results in accuracy. Having an ablation with a random subset of experts is helpful, but what could potentially happen, is that the specialised experts are a subset of the most frequently chosen experts. As a result, as we partially train the most frequently chosen experts, we can probably get some accuracy improvements as well (can be a future work suggestion).

So far we have only indirect evidence by comparing Jaccard overlaps between these domain-specific expert subset. At this point, can we estimate what is the overlap between the domain selected experts and the most frequently chosen ones? (clarification request)

  1. Tables and figures report average accuracies over Math, Medical and Remote Sensing domains. According to A5.2, Medical consists of PATHVQA, VQARAD and SLAKE; Math consists of MATHVISTA, MATHVISION, MMK12; Remote Sensing consists of HRVQA, RSVLMQA, RSVQALR and RSVQAHR. I could not find individual performance scores, so is it possible to report accuracies per dataset instead of averaged ones? Also, according to the line 1422, you use sampling in generation: is it possible to average the results across different random seeds to get more statistically significant results? (clarification request)

  2. How do you think this approach would generalise to pure LLMs (clarification request)? As far as I understand, VLM-related analysis is only present in modality specialisation study (e.g., Figure 1) in motivation section, but the domain specialisation in principle can be extended to pure LLMs (e.g., domains like math, coding, etc).

5 comments
AuthorsOpenReview ↗

Official Response to Reviewer NjBp

We thank the reviewer for their thoughtful and thorough evaluation of our work. We are happy to hear you found our “idea and method very clear” and appreciated the paper’s “rich insights and visualisations”. We address each of your comments below, and have incorporated the corresponding analyses and clarifications into the revised manuscript. Please let us know if you have any additional questions or comments.

I think it would be beneficial to additionally check how updating top ~20-40% most frequently chosen experts results in accuracy. [...] what could potentially happen, is that the specialised experts are a subset of the most frequently chosen experts [...] can we estimate what is the overlap between the domain selected experts and the most frequently chosen ones?

We agree that this is an important comparison. We kindly point the reviewer to the Frequency Selection baseline in Table 1, which we believe answers this question. For each domain, it trains the most frequently chosen experts, ranked by their routing counts on that domain's data, with the same expert budget as ExpertLens. This budget is 45% of all experts for math, 18% for medical, and 21% for remote sensing, which closely matches the 20-40% range you have suggested. This strategy for expert selection is entirely reasonable if we assume access to datasets for each domain, which is why we included it as a baseline. Our results show Frequency Selection performs well (56.0% mean accuracy), but ExpertLens matches it (56.5%) without using any domain data examples.

Note that, in practice, frequency selection is highly bounded by data availability and can become biased if the samples at selection time are not adequately representative of the entire domain. The time it takes to perform frequency selection also increases linearly with the number of samples used, so there is a tradeoff between selection quality and compute. ExpertLens is free from all of the above issues.

To determine if ExpertLens is simply selecting the most frequently routed experts, we can compute the amount of overlap between the Frequency Selection and ExpertLens experts.

DomainExpertsOverlapJaccard
Math2,7681,4320.35
Medical1,1031730.09
Remote sensing1,2902650.11

The overlap is minimal, especially for medical and remote sensing, so ExpertLens experts are not simply a subset of the most frequently chosen experts. We see instead that ExpertLens identifies an equally effective subset from the router weights alone without domain data. We also saw that matched-budget random subsets perform worse (Figure 7). Thus, it appears there are strategies for selecting the right experts. While Frequency Selection is a reasonable strategy, it requires domain data access, while ExpertLens is data-agnostic. The overlap analysis has been added to our manuscript in §A.3.

AuthorsOpenReview ↗

Official Response to Reviewer NjBp continued

Regarding the LoRA setup: the resulting model updates ~21-47% of model parameters, and LoRA training uses ranks 32 and 64. While I cannot directly estimate the percentage of updated parameters with these ranks, I assume it would be lower than what ExpertLense updates. As a result, the comparison is between a solid training setup for a part of the model weights vs updating (all?) experts but "a little bit" through a low LoRA rank training.

We would like to first clarify the purpose of each comparison and re-contextualize the baselines. First, the parameter-matched comparisons are the random expert subsets in Figure 7 (§4.2), Random (full) and Random (disjoint), together with the count-based selectors in Table 1, Frequency Selection and the Privileged Oracle. Each of these trains exactly the same number of experts as ExpertLens, with the same shared parameters, so they are the controlled comparisons for the question of which parameters to adapt.

We include LoRA at its standard ranks in our experiments as it is the standard parameter-efficient fine-tuning baseline and ExpertLens is itself an alternative recipe for parameter-efficient adaptation of MoE models. You are correct in your observations that at these ranks, LoRA “updates [fewer parameters] than ExpertLens”. However, matching LoRA to our parameter budget is not practical (or typical) for this model. With adapters on the attention projections and all expert MLPs, each unit of rank adds roughly 40M trainable parameters. Matching the 21.7-47.0% budgets of ExpertLens would require ranks of 165-357 (6.6B-14.3B adapter parameters). At these ranks, the adapter on the 768-dimensional expert MLPs approaches half of full rank and the method is no longer “low-rank”. Early in our experimentation, we did attempt higher ranks (small-scale runs at 128 and 256) and these runs collapsed early due to training instability.

We therefore view the comparisons as answering three separate questions. (1) ExpertLens versus FFT (Figure 6) asks whether selective adaptation of domain experts can match full fine-tuning with far less training time. (2) ExpertLens versus the random and count-based subsets (Figure 7 and Table 1) asks whether the identity of the trained experts matters, and if router-decoded selection identifies better parameters than data-free or data-driven alternatives at an identical budget. (3) ExpertLens versus LoRA (Figure 6) places the method in the context of standard PEFT: ExpertLens adds no new parameters and instead updates a targeted subset of existing ones. This comparison asks whether this structure-aware alternative is competitive with the default PEFT recipe at its conventional ranks. We thank the reviewer for this question and will clarify this distinction further in the updated manuscript.

Is it possible to report accuracies per dataset? [..] the tables should individually cover the datasets which were not present in training data.

Certainly! Thank you for the suggestion. We report full accuracies in §A.5.4 in the Appendix of the updated manuscript. Per your request, this table “individually cover[s] the datasets which were not present in the training data”. We clarify that there is no leakage in the training samples (between train and test splits). Additionally, we note that gains are not confined to the datasets used for training: every held-out benchmark (MathVista, MathVision, SLAKE, HRVQA, RSVLM-QA) improves after training on its domain, across all seeds. We report our results over three seeds (L355-356), with shaded regions showing 95% confidence intervals.

AuthorsOpenReview ↗

Official Response to Reviewer NjBp continued

How does this analysis generalise to pure LLMs?

This is a great question! In our work we primarily focus on MoE-VLMs, as prior work on language-only LLMs suggests experts are primarily clustered by part-of-speech (PoS). We discuss this briefly in (L094-097 and L237-239). Figure 3 reflects this clustering for text-token inputs to MoE-VLMs as well: text token activations were much more organized by PoS than by domain, contrary to image patch activations that are well-organized by domain.

However, to thoroughly explore your question for a language-only LLM, we repeat the KNN purity analysis on OLMoE, a language-only MoE, to see if text tokens are more organized by PoS than domain in an LLM-MoE as well. Our findings are below:

kGroupingEarly layersMiddle layersLate layers
3by dataset0.4220.4980.492
3by PoS0.7760.7360.692
5by dataset0.4030.4800.475
5by PoS0.7620.7220.677
7by dataset0.3900.4660.462
7by PoS0.7510.7110.666
15by dataset0.3520.4290.425
15by PoS0.7180.6810.636
30by dataset0.3150.3900.387
30by PoS0.6810.6480.604

We find that, across layers, the same pattern holds in OLMoE: text-only LLMs seem to organize their experts by part-of-speech. This is why it was exciting for us to find that semantic organization did emerge for image patches, unlike text tokens, in MoE VLMs! We hypothesize this may be due to the image patches having been processed through an image encoder before the downstream LLM, giving their encodings a semantic component at the starting layer of the LLM which text encodings do not receive.

If you are interested, we also point the reviewer to previous works exploring similar part-of-speech clustering effects for language-only MoEs. A particularly rich array of such experiments have been explored in the Mixtral of Experts paper (Jiang et. al, [1] - particularly Figures 7 and 8 in that paper).

Thank you for your suggestions and clarifications which improved the strength and clarity of our work! Below are our references:

[1] Jiang, Albert Q., et al. "Mixtral of experts." arXiv preprint arXiv:2401.04088 (2024).

Reviewer_rMbaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

ExpertLens uses router weights to choose which experts to fine-tune in a multimodal MoE. The language-model head converts each router vector into vocabulary scores; domain labels on the top tokens determine the selected experts. Training also updates shared decoder parameters. In sequential math, medical, and remote-sensing adaptation, the paper reports accuracy close to full fine-tuning with fewer updated parameters and less training time.

Critical strengths
  • Selection requires neither target-domain examples nor measurements of their routing. The resulting expert groups favor the corresponding domain inputs and compete with activation-based selectors during adaptation. This is the strongest evidence that the decoded labels can guide which parameters to train.
  • At the same expert budget, the selected subsets outperform unrestricted random selection. The shared-parameter ablations show why the recipe also updates the router, and the Kimi-VL experiment extends the adaptation finding beyond Qwen.
Critical weaknesses
  • The reported 4.0× speedup over full fine-tuning and 2.9× over LoRA need an explicit timing definition. Although data, steps, and hardware are matched, the paper does not specify which operations are timed or which modules receive LoRA. Different rollout lengths and implementation choices could change total time substantially. A breakdown is needed to distinguish savings in parameter updates from savings elsewhere in the training loop; otherwise, the contribution of selective expert updating remains unclear.
  • The TinyImageNet experiment holds dataset identity fixed, but visual similarity can still vary with semantic similarity. Related classes may share shape, texture, or background, which the mean-color and patch-position controls do not capture. The relationship between semantic distance and expert overlap could therefore reflect both semantics and appearance. The claim that it reflects semantics rather than visual similarity remains unresolved. This limits the explanation of expert specialization, even though the selected experts perform well in adaptation.
Questions for authors
  • Which operations are timed, what are the rollout lengths and per-stage costs, and which modules receive LoRA?
  • Does semantic distance still predict cross-class expert overlap after controlling for visual similarity?
5 comments
AuthorsOpenReview ↗

Official Response to Reviewer rMba

We thank the reviewer for their detailed and constructive assessment of our work. We found your questions and suggestions helpful. We address each of your comments below, and have incorporated the corresponding findings into the revised manuscript. Please let us know if you have any additional questions or comments.

Further details on training time comparisons

We thank you for this suggestion, which we agree would provide valuable context to the speedup provided by ExpertLens. We provide responses to your questions below. We have added the following details in the updated manuscript in §A.5.5.

Which operations are timed?

Wall-clock time in Fig. 6B is the per-step training time reported by the verl trainer, summed over the 200 steps of each stage. Each step comprises (i) rollout generation with vLLM (32 prompts × 5 samples), (ii) reward computation, (iii) the forward pass computing old log-probabilities, and (iv) the actor update with FSDP. All methods share the same trainer, data, number of steps, batch size, group size, and 4xH200 hardware.

What are the rollout lengths and per-stage costs? Are the benefits due to different rollout lengths? We have computed the rollout lengths and per-stage breakdown in the table below.

MethodStageGeneration (s)Old log-prob (s)Actor update (s)Total step (s)Mean response length (tokens)Stage wall-clock (h)
FFTMath22.828.11089116031264.5
FFTMedical8.024.61070111910762.2
FFTRemote sensing4.620.099210343257.4
ExpertLensMath17.618.630034248519.0
ExpertLensMedical3.716.41982248312.5
ExpertLensRemote sensing2.414.02052284612.7

The speedup comes from the actor update, not rollout. Rollout generation accounts for at most 2.0% of FFT step time and at most 5.1% of ExpertLens step time, and the old log-probability forward pass is very fast for both methods (14-28 s). Notably, rollout length doesn’t explain the advantage ExpertLens has over FFT. FFT step time is nearly flat (1034-1160 s) while its mean response length ranges from 32 to 312 tokens, and ExpertLens is faster on math despite generating responses 1.6× longer than FFT's. The vast difference in training time is concentrated in the actor update (992-1089s for FFT vs. 198-300 s for ExpertLens), where ExpertLens’ selective updates apply. Since unselected experts are detached in the backward pass, they receive no weight gradients and their inputs are never stored for the backward pass. Every other phase is shared between the two methods and costs roughly the same.

Which modules receive LoRA?

Adapters are applied to the attention projections (q, k, v, o) of every decoder layer and to the routed-expert MLP weights, following Wei et al., 2026 and Liu et al., 2026 (L105). We use rank 32 and α = 64 following Shuttleworth et al. 2025 (L373). Rank 64 was also evaluated but rank 32 performed better and is the configuration reported in the paper. These details have been added to §A.5.1 of our manuscript.

AuthorsOpenReview ↗

Official Response to Reviewer rMba continued.

TinyImageNet experiment holds dataset identity fixed, but visual similarity can still vary with semantic similarity. Related classes may share shape, texture, or background …

This is an interesting point. We’d like to first point to a few qualitative examples to illustrate how semantically similar classes can look visually distinct. We then share results from an additional ablation of the TinyImageNet experiment with a low-level visual similarity metric that goes beyond patch colors and indices. Our analysis shows semantics remain correlated with the number of experts shared between classes even when effects from visual similarity are accounted for.

In §3.3, we explored whether object level semantic specialization arises in MoE-VLMs. A quick visual intuition to answer your question is to refer to Figure 4B, which highlights image class pairs which are semantically similar, but visually distinct. For example, jellyfish and sea slugs look very different and live at different sea levels. Sea slugs are opaque and amorphous blobs of varying colors that get photographed in shallow, rocky waters, whereas jellyfish are transparent and can live in deep oceans with dark blue or black backgrounds. Tabby cats and golden retrievers are semantically similar because they are both pet animals, but one of them is gray and indoors, while the other is golden and often photographed outside. These visually distinct class pairs still exhibit a much higher degree of expert overlap than two randomly selected classes.

In addition to these examples, we can quantitatively address whether cross-class expert overlap remains after controlling for visual similarity by introducing a low level visual similarity metric. Specifically, we repeat the analysis in Figure 4A, but replace the WordNet distance metric with a low-level visual similarity metric to see how closely visual similarity is correlated to the expert-share Z score between class pairs. The new metric also enables us to compute the partial correlation between WordNet distance and expert-share Z score when low level visual similarity is held constant.

The low-level visual similarity metric we use here is feature similarity at the “stem” of ResNet50, or the global-average-pooled ReLU after the first convolution. We choose this metric because early CNN layers have been widely shown to encode low-level visual features (e.g. as found in Zeiler et. al, [1]).

The visual distance is then $1-cos(a,b)$ of two class prototypes a and b, so a lower value indicates the two classes are visually similar. We then compute the Pearson r for the new distance metric and the number of experts shared between classes, to see whether visual similarity is as strongly correlated to expert share Z-score as the WordNet distance. We also compute the partial Pearson r for the correlation for WordNet distance to expert share Z-score while visual similarity distance is held constant, and vice versa. We include the updated figure in the Appendix (§A.4.5), and the resulting Pearson r values below:

Predictor for ZPearson rPartial Pearson r
WordNet distance-0.380-0.358
ResNet-50 stem (low level visual similarity)-0.197-0.142

If we hold this visual similarity distance fixed, we find that the partial Pearson r for WordNet distance correlation to the class expert share Z-score only sees a minor drop from -0.380 to -0.358. In other words, semantic distance between classes is still correlated to the number of experts they share, even if we remove effects that could come from visual similarity between classes. This indicates that low level visual similarity does not explain the WordNet relationship away.

Further, the Pearson r for the visual distance as predictor for the expert-sharing Z score is -0.197 and drops to -0.142. This suggests that there was a smaller initial correlation between the number of experts shared between classes and their visual similarity, and that semantic similarity explains away a larger chunk of this correlation than vice versa. Thus, these experiments further reinforce our findings that experts are semantically organized.

Thank you for the suggestions! The requests for clarification and experiments have made our paper stronger, and we appreciate it. We put our references below.

[1] Zeiler, Matthew D., and Rob Fergus. "Visualizing and understanding convolutional networks." European conference on computer vision. Cham: Springer International Publishing, 2014.

Thank you again for taking the time to review our paper. As the discussion period deadline is approaching, we wanted to briefly follow-up to check if you had any remaining questions or comments. Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.