acceptodds
Under review as a conference paper at ICLR 2027

FA-MoE: Fisher-Adaptive Routing for Sparse Vision-Language Mixture-of-Experts

Abstract

Sparse Mixture-of-Experts (MoE) architectures have emerged as an efficient paradigm for scaling large vision-language models (LVLMs). In dense-to-MoE upcycling, experts inherit weights from a pretrained dense model, while the newly introduced router is commonly initialized randomly and trained with an auxiliary load-balancing objective. This leaves two gaps. The first arises at initialization: the experts inherit pretrained structure, whereas the router starts without information about the representation space it must partition. The second arises during training: auxiliary load balancing controls aggregate expert usage but does not directly regulate each token's routing distribution. We introduce Fisher-Adaptive MoE (FA-MoE) to address both gaps. FA-MoE initializes the router with Fisher-discriminative directions derived from the dense model's hidden representations, using K-means cluster assignments as pseudo-labels. During training, it combines token-level routing regularization that (1) controls probability mass outside the selected top-k experts and (2) adapts entropy within the selected set according to routing confidence, with batch-level regularization that maintains balanced expert utilization. Across StableLM-1.6B, Qwen-1.8B, and Phi-2 (2.7B) on eight multimodal benchmarks, FA-MoE ranks first in 21 of 24 backbone-benchmark settings. Controlled ablations show that Fisher initialization performs better than random initialization under the full routing objective, while routing analyses show fewer subsequent changes in the selected expert set.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision-langauge models, Mixture-of-Experts

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_m3b3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FA-MoE improves dense-to-MoE upcycling through representation-based router initialization and token-level routing regularization. It derives Fisher directions from clustered dense-model activations and combines leakage, adaptive entropy, and utilization losses during training. Evaluation across three vision-language backbones and eight benchmarks shows broadly improved performance, supported by initialization ablations and routing analyses.

Critical strengths
  1. The initialization contribution is supported by a relevant controlled comparison. With the cosine router and training objective fixed, Fisher initialization outperforms centroid initialization on all five metrics in Table 2, indicating value beyond simply using clustered dense representations.

  2. The empirical improvements extend across three backbones and multiple benchmarks. In particular, gains over V-MoE are consistent on MME and SEED-Bench, while the reported three-seed FA-MoE results support the stability of the complete method.

Critical weaknesses
  1. Statistical support for the component claims is limited. Table 2 contains single-run comparisons on StableLM, and Appendix C reports variability only for FA-MoE. This leaves the robustness of the initialization gains and several small advantages over competing methods uncertain.

  2. The necessity of adaptive entropy is not established. Table 2 omits leakage plus utilization without entropy, while Table 6 compares adaptive entropy only with ordinary entropy minimization. These comparisons do not show whether adaptive entropy improves upon omitting the entropy term altogether.

Questions for authors
  1. Do the matched initialization and objective gains in Table 2 persist across seeds, and what is the variability of the principal baselines?
  2. With Fisher initialization and all other settings fixed, how does leakage plus utilization without an entropy term compare with the full objective?
  3. The comparison methods are too outdated; please compare against the latest baselines.
2 comments
AuthorsOpenReview ↗

Response to Reviewer m3b3

Thank you for the careful review. Below we report the baseline seed variability and an ablation that removes the entropy term while keeping leakage and utilization, together with a correction to the StableLM POPE standard deviation in Table 5.

W1/Q1: Seed variability of the baseline, and robustness of the Table 2 gains. Three-seed results (mean ± std) for V-MoE (second best method overall), next to FA-MoE (Table 5):

BackboneMethodGQASQA-ITextVQAPOPEMME
StableLMV-MoE60.5±0.359.6±0.249.8±0.386.1±0.21544.2±14.8
StableLMFA-MoE61.4±0.160.0±0.150.7±0.286.4±0.51597.9±8.7
QwenV-MoE61.6±0.361.5±0.448.8±0.586.9±0.21513.0±11.4
QwenFA-MoE61.6±0.262.8±0.348.9±0.387.2±0.21562.4±9.1
Phi-2V-MoE61.6±0.370.6±0.452.7±0.586.3±0.21598.5±22.4
Phi-2FA-MoE61.7±0.271.4±0.353.0±0.387.3±0.11661.4±16.3

MME improves on all three backbones by roughly 50–63 points, well beyond the seed standard deviations (8.7–22.4), and SQA-I improves on all three backbones. We read the smaller differences (Qwen GQA, TextVQA and POPE; Phi-2 GQA and TextVQA; StableLM POPE) conservatively, as they are comparable to the seed variation. V-MoE's mean is not above FA-MoE's on any of the 15 pairs.

Table 2 reports single runs, with Fisher+Full being one of the three FA-MoE seeds above; we have not yet rerun its other configurations with multiple seeds and will do so for the revised paper. What we can offer now is a different kind of robustness: the Fisher-over-centroid ordering holds in every new matched configuration we trained, with 5K and 10K initialization examples (+1.20 and +1.16 GQA), eight experts (+0.8 GQA, +16.7 MME), and against PCA, a modality-aware design and a linear classifier trained on the same pseudo-labels (61.47 vs. 60.83 GQA). At 2K examples the gap narrows to +0.18 GQA, which we do not count as evidence either way. These runs share data and training setup, so they are not independent replicates and do not replace paired seeds. What they do show is that the Fisher advantage is not specific to one sampling budget, expert count or comparison baseline.

W2/Q2: Leakage + utilization without an entropy term. With Fisher initialization and everything else fixed, the three options for the entropy term compare as follows:

Entropy term (with leakage + utilization)GQASQA-ITextVQAPOPEMME
None (new)60.4059.7850.1086.011568.00
Ordinary entropy minimization (Table 6)57.2057.3047.9085.901558.10
Margin-aware adaptive (full)61.4759.9250.8686.931594.28

Removing the entropy term lowers all five metrics (−1.07 GQA, −26.28 MME), and ordinary entropy minimization is worse than having no entropy term at all (−3.20 GQA, −9.90 MME).

The reason is that leakage and utilization decide which experts are selected and how usage is balanced, but not how the two selected experts share a token; that is the entropy term's job. Both selected experts run for every token and are mixed with the weights p^(1)\hat p_{(1)} and p^(2)\hat p_{(2)} (Eqs. 3–4). Ordinary entropy minimization pushes every token toward a sharp split, whether its routing is confident or not, so the second expert's weight, and the gradient it receives from that token, shrinks. For nearly tied tokens, this forces an early commitment to one expert despite the small margin (Sec. 2.3). The adaptive version sharpens only high-margin tokens and lets nearly tied tokens keep both experts, which is where the router is least certain.

Q3: Choice of baselines. We chose baselines by what FA-MoE changes rather than by date. The method touches three things (router geometry, dense-to-sparse upcycling and representation-aware initialization), and Table 1 has a strong representative of each: X-MoE and StableMoE for routing, Drop-Upcycling (Nakamura et al., 2025) for upcycling, and Cluster-Aware Upcycling (CA-MoE; Chu et al., 2026) for initialization. CA-MoE is the closest method; FA-MoE is higher in all 24 settings of Table 1 and on all four LLaVA-1.5-7B metrics we added (64.19 vs. 62.69 GQA; 1626.11 vs. 1591.43 MME). We have not run every recent routing method. The revised paper will say which families we cover and why, and will discuss ProMoE (Wei et al., ICLR 2026), which another reviewer pointed us to. If there is a specific method you consider a missing baseline, we would be glad to know which.

Dear Reviewer,

Thank you again for your reviews. We have posted the response with the additional experiments and clarifications requested, and we would welcome your feedback on whether they address your concerns. If anything remains unclear, or you would like another experiment, we are glad to respond before the discussion period ends.

We appreciate your time.

Reviewer_wW6yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Fisher-Adaptive MoE (FA-MoE) for converting pretrained dense vision-language models into sparse mixture-of-experts models. It first applies K-means to dense hidden representations and then uses the cluster assignments as pseudo-labels to compute Fisher-discriminative directions for router initialization. The experts are initialized by copying the original dense feed-forward networks. During training, the method combines three regularization terms: reducing probability mass assigned to unselected experts, adapting entropy within the selected set according to the gap between the two highest routing probabilities, and regularizing batch-averaged expert routing probabilities. Experiments on three compact vision-language backbones and eight benchmarks report the highest score.

Critical strengths

Building on the existing idea of using dense-model representations to initialize MoE models, the paper proposes Fisher-discriminative initialization combined with three routing objectives. Within the reported experimental scope, the combined method outperforms the compared methods in most settings.

Critical weaknesses
  1. It remains insufficiently established whether the advantage of Fisher initialization over centroid initialization depends on the current sampling scale and expert configuration. The sample size, representativeness, and the distribution of samples across clusters may affect both methods. The observed difference has been established under particular sampling conditions. With more samples, direct centroid initialization may improve, potentially changing the gap between the two approaches.
  2. The motivation and contribution relative to Cluster-Aware Upcycling remain insufficiently justified. That work already uses dense activation structure to initialize sparse routing and provides training guidance for uncertain routing. The geometric distinction between cluster centroids and Fisher-discriminative directions does not yet adequately explain the specific limitation of centroid-based routing in the present task, or why improving the separability of pseudo-labels should lead to better expert assignments.
Questions for authors

The full routing objective is motivated by token-level routing uncertainty that can also arise in ordinary MoE training. Why does it fail to improve most metrics under random initialization?

3 comments
AuthorsOpenReview ↗

Response to Reviewer wW6y (1/2)

Thank you for the review. All new runs below use StableLM-1.6B, the same cosine router, the full objective (leakage + adaptive entropy + utilization) and identical training, changing only the factor under study.

W1: Does the advantage over centroid initialization depend on sampling scale and expert configuration? Partly: there is some sensitivity at the smallest sample size. Here, 5K means 5,000 image–text examples, each contributing many token representations per layer.

Init. examplesGQA CentroidGQA FisherMME CentroidMME Fisher
2K59.1259.301548.151551.47
5K (Table 2(a))60.2761.471566.281594.28
10K60.3561.511568.311595.13

At 2K the gap narrows to 0.18 GQA / 3.3 MME, too small to support a conclusion from a single run; at 5K and 10K it is 1.20 / 28.0 and 1.16 / 26.8, and the centroid baseline itself moves by only +0.08 GQA between 5K and 10K. A plausible reason for the 2K result is a less reliable Fisher estimate from fewer tokens per cluster. The 5K estimate, by contrast, is stable according to App. H.7: Fisher subspaces estimated from two disjoint 5K-example subsets overlap strongly (0.77) and remain discriminative on held-out data (J = 7.34 vs. 0.37 for a random subspace of the same rank; layer 0, five repetitions). On representativeness, the vision/text mix of the sample is almost identical at all three sizes (about 90.7% visual tokens).

For the expert configuration, we repeated the comparison with eight experts (top-2, single run):

Initialization (full objective)GQAMME
Centroid60.71586.9
Fisher (ours)61.51603.6

Fisher stays ahead (+0.8 GQA, +16.7 MME), a smaller margin than with four experts. App. D.2 shows the same ordering against V-MoE at eight experts.

W2: Contribution relative to Cluster-Aware Upcycling. CA-MoE and FA-MoE start from the same raw material, dense-model token representations and a K-means partition of them, and diverge immediately after. CA-MoE initializes the router from K-means centroids, builds cluster-specific experts with truncated SVD and adds expert-ensemble self-distillation. We use the K-means labels only to compute Fisher directions, keep the experts as identical copies, and regularize each token's routing distribution without a teacher.

The distinction matters for our router, which scores a token by cosine similarity to each router row (Eq. 8). A normalized centroid is the direction to a cluster's center: it tells the router where the cluster is, not what separates it from the others (App. H.4). Fisher uses the same pseudo-labels but extracts the directions that distinguish the groups. The difference is visible in the initial routing: K-means puts 87–100% of text tokens into one cluster in most layers, and the centroid-initialized router sends most text tokens to one expert, whereas the Fisher-initialized router spreads them over two to three experts. This matters in upcycling because the copied experts start identical, so early in training the task loss says little about routing, and the initial routing decides which token updates each expert receives (Sec. 1).

The Centroid+Full control in Table 2(a) is not the complete CA-MoE method: it changes only the initialization and keeps our router and objective, without CA-MoE's SVD experts or distillation, so it isolates the effect of initialization. In this matched setting, Fisher+Full beats Centroid+Full on all five metrics (+1.20 GQA, +28.00 MME). It also beats a linear classifier trained on the same pseudo-labels (61.47 vs. 60.83 GQA) and a modality-aware initialization that clusters vision and text tokens separately (61.47 vs. 60.62 GQA). Routing is also more stable: 7.4% of tokens change their top-2 set between consecutive checkpoints (100 steps apart), against 9.2% for Centroid+Full. Comparing the complete methods, FA-MoE is higher than CA-MoE in all 24 settings of Table 1 and on LLaVA-1.5-7B (64.19 vs. 62.69 GQA; 1626.11 vs. 1591.43 MME).

AuthorsOpenReview ↗

Response to Reviewer wW6y (2/2)

Q: Interaction between initialization and the full routing objective. Initialization sets the initial token partition, and the objective shapes how that partition evolves. Each term acts on the router's current partition: leakage keeps probability on the selected experts, adaptive entropy sets how the two selected experts share a token, and utilization balances usage. None of them says which tokens belong together, and in upcycling the task loss gives little signal about this early on: at initialization the experts are identical and the selected weights sum to one (Eq. 3), so the MoE output does not depend on how probability is split between them, and the task loss gives the router no gradient. The regularizers, however, act from the first step. With random initialization, experts are interleaved within the same regions (Fig. 1(b)), so the objective sharpens a partition that does not follow the representation structure; with Fisher initialization, it sharpens one that already does (Fig. 1(c)).

Table 2(a) shows this. Switching from the auxiliary to the full objective lowers four of five metrics with random initialization but raises four of five with Fisher initialization (MME: −7.53 vs. +29.45). The flip rates agree: with the full objective, 0.101 under random and 0.074 under Fisher initialization. So the two components interact rather than add: the same objective helps from a Fisher start and hurts from a random one. The mechanism above is our explanation of that pattern; the metrics and flip rates are consistent with it, but they do not prove it.

We will add the sampling, eight-expert and stability results to the revised paper.

Dear Reviewer,

Thank you again for your reviews. We have posted the response with the additional experiments and clarifications requested, and we would welcome your feedback on whether they address your concerns. If anything remains unclear, or you would like another experiment, we are glad to respond before the discussion period ends.

We appreciate your time.

Reviewer_ggEJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose FA-MoE, which collects hidden states from target layers, generates pseudo-classes via K-means, and uses Fisher LDA to derive discriminative directions for router initialization. This addresses the structural asymmetry in dense-to-MoE upcycling, where experts inherit pretrained weights while the router is randomly initialized. FA-MoE further introduces three losses: a routing leakage loss that reduces probability mass outside the top-k experts, margin-aware adaptive entropy that adjusts the probability distribution within the top-k set based on routing confidence, and batch-level utilization regularization that balances expert usage. Experiments across multiple backbones and multimodal benchmarks demonstrate the effectiveness of the proposed method.

Critical strengths
  • The paper addresses an important problem: the structural asymmetry caused by random router initialization.
  • FA-MoE is simple and lightweight, and works synergistically with the proposed routing losses to improve model performance.
Critical weaknesses
  • It remains unclear whether Fisher initialization is the best strategy. Alternative approaches, such as a lightweight MLP, modality-aware initialization, PCA, or spectral clustering, may further improve performance.
  • Prior work has explored clustering- or semantics-guided MoE routing, such as Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance. Although it targets DiTs, it is closely related and should be discussed.
  • The gains over the baseline are modest. For example, on Qwen in Table 1, improvements on five of the eight benchmarks are below one point.
  • Fisher initialization relies on only approximately 5K tokens for pseudo-label generation, but lacks downstream ablations on sample size and the image–text token distribution.
  • All three backbones are relatively small (1.6B–2.7B). Experiments on larger open-source models are needed to demonstrate scalability.
Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ggEJ (1/2)

Thank you for the review and the specific suggestions. We ran the PCA, modality-aware and linear-classifier initializations, the sampling-scale sweep and a 7B backbone; spectral clustering and an MLP router we discuss rather than run, for the reasons given under W1. Unless noted, all runs use StableLM-1.6B, four copied experts with top-2 routing, the same cosine router, the same 5,000 initialization examples, the full objective and identical training.

W1: Is Fisher initialization the best choice? We compared the suggested alternatives in the same setting:

InitializationInformation usedGQASQA-IMME
RandomNone59.2359.391547.57
PCA (top-4 directions)Variance only58.6458.121556.23
K-means centroidsK-means labels60.2758.151566.28
Modality-aware (2+2)Modality + K-means60.6258.211571.42
Linear classifierK-means labels60.8359.231571.00
Fisher (ours)K-means labels61.4759.921594.28

Fisher is best in every column. PCA uses the same representations but picks the directions of largest variance, and it is below random initialization on GQA and SQA-I; a high-variance direction need not separate groups of tokens. Our modality-aware variant runs K-means with K=2 separately on vision and text tokens (routing is not restricted by modality during training). It stays close to plain centroids and below Fisher on all three metrics; with about 91% of tokens visual, a fixed 2+2 split by modality is a restrictive allocation, though other modality-aware designs could behave differently. The linear classifier is trained on the same pseudo-labels as Fisher; it improves on centroids but stays below Fisher.

Spectral clustering would replace K-means as the source of pseudo-labels; Fisher directions can be computed from either, so it is a complement rather than an alternative, and we leave it as a natural follow-up. An MLP router changes the router rather than its initialization, which is the variable under study here. Within the cosine-router family, Fisher is the best of the six initializations we have now tested.

W2: Relation to "Routing Matters in MoE" (ProMoE). Thank you for the pointer; we will cite and discuss it. ProMoE and FA-MoE share the view that routing should follow representation structure, and both score tokens by cosine similarity to one vector per expert. The settings and mechanisms differ. ProMoE trains Diffusion Transformers from scratch for class-conditional image generation, learns expert prototypes jointly with the network, and shapes them with a contrastive loss between each prototype and the tokens assigned to it (after a diffusion-specific step that sends null-conditioned tokens to separate experts). FA-MoE upcycles a pretrained vision-language model, where the dense model already provides structured representations before sparse training starts. We compute the initial router weights once from those representations (they remain trainable) and regularize each token's routing distribution during training. ProMoE's finding that clustering-based guidance helps vision MoE fits our observation that directions separating the groups initialize the router better than centroids (W1). The two ideas could also be combined: in an upcycling version of ProMoE, Fisher directions could initialize the prototypes.

W3: Size and consistency of the gains. Three points. First, the gains are consistent: FA-MoE's mean exceeds V-MoE's in 22 of the 24 settings of Table 1, is equal in one (Qwen GQA), and is lower in one (Phi-2 MMBench). Its mean rank across eight benchmarks and three backbones is 1.10, against 2.40 for V-MoE, the next-best method. Second, the largest gains exceed run-to-run variation: with three seeds for both methods, MME improves by 53.7, 49.4 and 62.9 points on StableLM, Qwen and Phi-2, against seed standard deviations of 8.7–22.4, and SQA-I improves on all three backbones. Not every difference clears that bar: on Qwen, five of the eight are below one point, and we treat those (GQA among them) as ties rather than wins. The claim rests on MME and SQA-I and on the direction being the same almost everywhere. Third, the added cost is small: one-time initialization takes about 12.5 minutes for StableLM-1.6B and 30 minutes for Phi-2 (App. H.9), and inference keeps the baseline's top-2 computation.

AuthorsOpenReview ↗

Response to Reviewer ggEJ (2/2)

W4: Sample size and image–text distribution. Here, 5K means 5,000 image–text examples. We varied the number of initialization examples with all other settings fixed:

Init. examplesGQA CentroidGQA FisherMME CentroidMME Fisher
2K59.1259.301548.151551.47
5K (Table 2(a))60.2761.471566.281594.28
10K60.3561.511568.311595.13

At 2K both initializations are worse and the gap narrows to 0.18 GQA / 3.3 MME. A likely reason is that 2K examples give a less reliable estimate of the Fisher subspace, which depends on within-cluster scatter; we have not tested this directly. At 5K and 10K the gap is similar (1.20 GQA / 28.0 MME and 1.16 / 26.8), and App. H.7 shows that Fisher subspaces from two disjoint 5K subsets overlap strongly (0.77). In our setting 5K examples appear sufficient, and going to 10K adds little for either initialization (centroid: +0.08 GQA, +2.03 MME).

On the image–text distribution, the sampled tokens are about 90.7% visual and 9.3% text at all three sizes. K-means places 87–100% of the text tokens in a single cluster in most layers. In the initial routing, the centroid-initialized router sends most text tokens to one expert, whereas the Fisher-initialized router spreads them over two to three experts. The sweep therefore varies sample count at a fixed image–text ratio; we did not vary the ratio itself, and we will state this scope in the revised paper.

W5: Scalability to larger backbones. We ran LLaVA-1.5-7B with the same upcycling setup (four experts, top-2) for all three methods:

MethodGQASQA-IMME
V-MoE63.2167.131583.22
CA-MoE62.6966.231591.43
FA-MoE (ours)64.1968.721626.11

FA-MoE is best on all three metrics: +0.98 GQA, +1.59 SQA-I and +42.89 MME over V-MoE, and +1.50, +2.49 and +34.68 over CA-MoE. This carries the 1.6B result over to a 7B backbone, though two model sizes do not establish a scaling trend. These are single runs, so we read them as a consistency check at the larger scale rather than a statistical comparison.

Dear Reviewer,

Thank you again for your reviews. We have posted the response with the additional experiments and clarifications requested, and we would welcome your feedback on whether they address your concerns. If anything remains unclear, or you would like another experiment, we are glad to respond before the discussion period ends.

We appreciate your time.

Reviewer_p2DoOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FA-MoE for dense-to-MoE upcycling in vision-language models. It initializes routers using K-means and Fisher discriminant analysis on pretrained representations, then combines leakage reduction, confidence-adaptive entropy, and expert balancing. Experiments on three backbones and eight multimodal benchmarks report the best result in 21 of 24 settings and improved routing stability.

Critical strengths
  1. The paper addresses the important problem of exploiting pretrained representation structure during dense-to-MoE upcycling rather than randomly initializing routers.
  2. Fisher initialization is compared with random and centroid initialization under controlled settings, providing direct evidence for the proposed initialization.
  3. The evaluation covers three backbones and eight benchmarks, with additional analyses of routing stability, utilization, and leakage.
Critical weaknesses
  1. Representation-aware router initialization has been explored in recent work, so the main novelty is relatively incremental and relies on combining Fisher initialization with existing regularization ideas.
  2. Fisher LDA yields discriminative directions rather than class-specific expert assignments. The paper does not explain why these directions should be used directly as router weights or show stronger specialization than a classifier trained from the same pseudo-labels.
  3. For top-2 routing, the top-2/top-3 margin appears more relevant than the chosen top-1/top-2 margin. Moreover, leakage is already below 0.01 at initialization, leaving the independent contribution of the leakage loss unclear.
Questions for authors
  1. Why are Fisher discriminative directions used directly as expert router weights? Could the authors compare against a classifier trained using the same K-means pseudo-labels?
  2. Why does the adaptive entropy objective use the top-1/top-2 margin instead of the top-2/top-3 margin? How would this alternative affect routing stability and downstream performance?
  3. Given that routing leakage is already very low at initialization, what additional benefit does the leakage loss provide? Could the authors report leakage trajectories and performance under matched objectives?
  4. Could the authors provide matched routing-stability comparisons across Random+Full, Centroid+Full, Fisher+Auxiliary, and Fisher+Full to isolate the contribution of each component?
3 comments
AuthorsOpenReview ↗

Response to Reviewer p2Do (1/2)

Thank you for the positive review and the concrete questions. We ran all four requested controls. All runs use StableLM-1.6B, four copied experts, top-2 routing, the same cosine router and the same training setup, and change only the component in question.

W1: Novelty. The closest prior work is Cluster-Aware Upcycling (CA-MoE), and the distinction is worth stating precisely. CA-MoE initializes the router from K-means centroids, builds cluster-specific experts by truncated SVD, and adds self-distillation. FA-MoE uses the K-means labels only to estimate Fisher discriminative directions for the router, keeps the experts as identical copies, and shapes each token's routing distribution with a margin-aware objective. The contribution is the choice of discriminative rather than locational directions as the initial router, and its interaction with the objective. Three controlled results support this. First, using the same representations and pseudo-labels, Fisher outperforms centroid initialization on all five metrics (Table 2(a)) and a linear classifier trained on the same labels (Q1 below). Second, the interaction with the routing objective matters: the full objective improves four of five metrics with Fisher initialization, whereas it lowers four of five with random initialization (Table 2(a)). Third, the complete FA-MoE method outperforms CA-MoE in all 24 settings of Table 1.

Q1: Why use Fisher directions as router weights, and how do they compare with a classifier trained on the same pseudo-labels? Our router is a cosine router (Eq. 8):

rt,i=s⟨xt∥xt∥2,wi∥wi∥2⟩,r_{t,i} = s\left\langle \frac{x_t}{\|x_t\|_2}, \frac{w_i}{\|w_i\|_2} \right\rangle,

so each router row is a direction, and a token is sent to the experts whose rows it aligns with most. Because the copied experts are identical at the start, these directions decide which tokens, and therefore which gradients, each expert receives. We therefore want rows that separate the groups already present in the dense representations. A normalized centroid describes where a cluster is, whereas Fisher directions explicitly seek separation among the pseudo-labeled groups (App. H.4). The K-means labels define groups from which we estimate discriminative directions; they do not impose a one-cluster-per-expert assignment. For example, K-means places most text tokens in a single cluster in most layers, whereas the Fisher-initialized router spreads them over two to three experts.

For the classifier control, we trained a linear classifier on the same K-means labels and used its weights as the initial router:

Initialization (full objective)GQASQA-IMME
Centroid (Table 2(a))60.2758.151566.28
Linear classifier, same pseudo-labels60.8359.231571.00
Fisher (Table 2(a))61.4759.921594.28

The classifier improves on centroids but stays below Fisher (−0.64 GQA, −0.69 SQA-I, −23.28 MME). With the pseudo-labels held fixed, the discriminative directions are what make the difference.

Q2: Why the top-1/top-2 margin instead of the top-2/top-3 margin? The two margins measure different things. Adaptive entropy acts on the renormalized weights of the two selected experts, p^(1)=p(1)/(p(1)+p(2))\hat p_{(1)} = p_{(1)}/(p_{(1)}+p_{(2)}) and p^(2)=p(2)/(p(1)+p(2))\hat p_{(2)} = p_{(2)}/(p_{(1)}+p_{(2)}) (Eqs. 3, 10), so the quantity it regulates depends only on how p(1)p_{(1)} compares with p(2)p_{(2)}, and the top-1/top-2 margin is the matching confidence signal. Whether a third expert is close to entering the top-2 set is already handled by the leakage term (Eq. 9), which penalizes all probability outside the selected experts.

We tested the alternative by substituting the top-2/top-3 margin directly into αt\alpha_t, with γ\gamma and all other settings unchanged:

Margin in αt\alpha_tGQASQA-IMME
Top-2/top-357.5153.471501.79
Top-1/top-2 (ours)61.4759.921594.28

The direct substitution reduces performance (−3.96 GQA, −6.45 SQA-I, −92.49 MME), and the margin geometry explains why. Leakage is 1−p(1)−p(2)1 - p_{(1)} - p_{(2)}, so p(3)p_{(3)} cannot exceed the leakage, which stays below 0.01 (App. E.1). The top-2/top-3 margin is then roughly p(2)≈(1−mt)/2p_{(2)} \approx (1 - m_t)/2, where mtm_t is the top-1/top-2 margin, so it moves opposite to mtm_t. A nearly tied token (p(1)≈p(2)≈0.5p_{(1)} \approx p_{(2)} \approx 0.5) gets a large margin and is pushed to sharpen, while a confident token gets a margin near zero and is pushed to split its weight: the reverse of what the loss is meant to do. This concerns the matched, direct substitution; a separately calibrated top-2/top-3 objective would be a different design.

AuthorsOpenReview ↗

Response to Reviewer p2Do (2/2)

Q3: Leakage is already below 0.01 at initialization. What does the leakage loss add? Low initial leakage is expected from the Fisher initialization and cosine router (App. E.1). The leakage loss matters during training: as the router and experts evolve, it constrains the probability mass on unselected experts, which neither the task loss nor the batch-level utilization term regulates directly. To isolate it, we trained Adaptive + Utilization with and without the leakage term from the same Fisher initialization:

ObjectiveLeakage ↓Top-2 flip rate ↓MME ↑
Adaptive + Utilization0.0200.0911544.36
Full (+ Leakage)0.0040.0741594.28

Leakage and flip rate are averaged over training checkpoints; the flip rate is the fraction of tokens whose top-2 set changes between consecutive checkpoints, 100 steps apart. Without the leakage term, leakage is about 5× higher, the top-2 set changes more often, and MME drops by 49.92 points; all five metrics drop (Table 2(b), e.g., GQA 61.47 → 59.69). We will add the per-checkpoint leakage trajectories for both runs to the revised appendix.

Q4: Matched routing-stability comparisons. Tables 10 and 11 compare the complete V-MoE and FA-MoE systems, so we computed the top-2 flip rate for the four matched settings:

Matched settingFlip rate ↓MME ↑
Random + Full0.1011547.57
Centroid + Full0.0921566.28
Fisher + Auxiliary0.1861564.83
Fisher + Full0.0741594.28

With the objective fixed, the flip rate goes from 0.101 (random) to 0.092 (centroid) to 0.074 (Fisher), i.e., 27% and 20% fewer flips. With Fisher initialization fixed, the full objective reduces flips from 0.186 to 0.074 compared with the auxiliary loss, which balances usage across the batch but does not act on each token's distribution. (Fisher + Auxiliary uses the cosine router, so its value is not comparable to the linear-router V-MoE in Table 10.)

We agree that the flip rate measures routing stability, not specialization: a router can be stable and still uninformative. We do not claim learned semantic specialization, and the paper avoids that claim (Sec. 4, App. F). What we can show after training is narrower: in the trained FA-MoE, all four experts stay in use, and they receive different proportions of text and image tokens without any modality assignment (Fig. 3; App. F, Figs. 5–7). With the objective fixed, Fisher initialization gives a more structured initial partition (Fig. 1; App. J for three backbones at four depths) and fewer routing changes during training. All four controls will be added to the revised paper.

Dear Reviewer,

Thank you again for your reviews. We have posted the response with the additional experiments and clarifications requested, and we would welcome your feedback on whether they address your concerns. If anything remains unclear, or you would like another experiment, we are glad to respond before the discussion period ends.

We appreciate your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.