- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
FA-MoE improves dense-to-MoE upcycling through representation-based router initialization and token-level routing regularization. It derives Fisher directions from clustered dense-model activations and combines leakage, adaptive entropy, and utilization losses during training. Evaluation across three vision-language backbones and eight benchmarks shows broadly improved performance, supported by initialization ablations and routing analyses.
Critical strengths
-
The initialization contribution is supported by a relevant controlled comparison. With the cosine router and training objective fixed, Fisher initialization outperforms centroid initialization on all five metrics in Table 2, indicating value beyond simply using clustered dense representations.
-
The empirical improvements extend across three backbones and multiple benchmarks. In particular, gains over V-MoE are consistent on MME and SEED-Bench, while the reported three-seed FA-MoE results support the stability of the complete method.
Critical weaknesses
-
Statistical support for the component claims is limited. Table 2 contains single-run comparisons on StableLM, and Appendix C reports variability only for FA-MoE. This leaves the robustness of the initialization gains and several small advantages over competing methods uncertain.
-
The necessity of adaptive entropy is not established. Table 2 omits leakage plus utilization without entropy, while Table 6 compares adaptive entropy only with ordinary entropy minimization. These comparisons do not show whether adaptive entropy improves upon omitting the entropy term altogether.
Questions for authors
- Do the matched initialization and objective gains in Table 2 persist across seeds, and what is the variability of the principal baselines?
- With Fisher initialization and all other settings fixed, how does leakage plus utilization without an entropy term compare with the full objective?
- The comparison methods are too outdated; please compare against the latest baselines.
2 comments
Dear Reviewer,
Thank you again for your reviews. We have posted the response with the additional experiments and clarifications requested, and we would welcome your feedback on whether they address your concerns. If anything remains unclear, or you would like another experiment, we are glad to respond before the discussion period ends.
We appreciate your time.
Response to Reviewer m3b3
Thank you for the careful review. Below we report the baseline seed variability and an ablation that removes the entropy term while keeping leakage and utilization, together with a correction to the StableLM POPE standard deviation in Table 5.
W1/Q1: Seed variability of the baseline, and robustness of the Table 2 gains. Three-seed results (mean ± std) for V-MoE (second best method overall), next to FA-MoE (Table 5):
MME improves on all three backbones by roughly 50–63 points, well beyond the seed standard deviations (8.7–22.4), and SQA-I improves on all three backbones. We read the smaller differences (Qwen GQA, TextVQA and POPE; Phi-2 GQA and TextVQA; StableLM POPE) conservatively, as they are comparable to the seed variation. V-MoE's mean is not above FA-MoE's on any of the 15 pairs.
Table 2 reports single runs, with Fisher+Full being one of the three FA-MoE seeds above; we have not yet rerun its other configurations with multiple seeds and will do so for the revised paper. What we can offer now is a different kind of robustness: the Fisher-over-centroid ordering holds in every new matched configuration we trained, with 5K and 10K initialization examples (+1.20 and +1.16 GQA), eight experts (+0.8 GQA, +16.7 MME), and against PCA, a modality-aware design and a linear classifier trained on the same pseudo-labels (61.47 vs. 60.83 GQA). At 2K examples the gap narrows to +0.18 GQA, which we do not count as evidence either way. These runs share data and training setup, so they are not independent replicates and do not replace paired seeds. What they do show is that the Fisher advantage is not specific to one sampling budget, expert count or comparison baseline.
W2/Q2: Leakage + utilization without an entropy term. With Fisher initialization and everything else fixed, the three options for the entropy term compare as follows:
Removing the entropy term lowers all five metrics (−1.07 GQA, −26.28 MME), and ordinary entropy minimization is worse than having no entropy term at all (−3.20 GQA, −9.90 MME).
The reason is that leakage and utilization decide which experts are selected and how usage is balanced, but not how the two selected experts share a token; that is the entropy term's job. Both selected experts run for every token and are mixed with the weights p^(1) and p^(2) (Eqs. 3–4). Ordinary entropy minimization pushes every token toward a sharp split, whether its routing is confident or not, so the second expert's weight, and the gradient it receives from that token, shrinks. For nearly tied tokens, this forces an early commitment to one expert despite the small margin (Sec. 2.3). The adaptive version sharpens only high-margin tokens and lets nearly tied tokens keep both experts, which is where the router is least certain.
Q3: Choice of baselines. We chose baselines by what FA-MoE changes rather than by date. The method touches three things (router geometry, dense-to-sparse upcycling and representation-aware initialization), and Table 1 has a strong representative of each: X-MoE and StableMoE for routing, Drop-Upcycling (Nakamura et al., 2025) for upcycling, and Cluster-Aware Upcycling (CA-MoE; Chu et al., 2026) for initialization. CA-MoE is the closest method; FA-MoE is higher in all 24 settings of Table 1 and on all four LLaVA-1.5-7B metrics we added (64.19 vs. 62.69 GQA; 1626.11 vs. 1591.43 MME). We have not run every recent routing method. The revised paper will say which families we cover and why, and will discuss ProMoE (Wei et al., ICLR 2026), which another reviewer pointed us to. If there is a specific method you consider a missing baseline, we would be glad to know which.