acceptodds
Withdrawn from ICLR 2027

Routing by Need, Not by Modality: Demand-Conditioned Sparse Activation for MoE-VLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mixture-of-Experts (MoE) has become the default backbone for large vision-language models (VLMs), yet how modality signals should govern expert routing remains unsettled: existing strategies are hand-crafted or modality-agnostic, and recent modality-guided methods score each token by *which modality it belongs to*. We argue this is the wrong quantity — analyzing token-level fusion across layers, we find that modality identity and computational need are only weakly coupled, so routing by identity spends capacity where it is least needed. We propose **CoDA-MoE**, which routes by cross-modal information demand instead: a cheap single-pass probe estimates a per-token demand score measuring how much a token's representation still depends on the complementary modality; a demand-conditioned activation budget lets the number of activated experts vary per token under a global sparsity constraint, actively down-routing low-demand tokens and redirecting capacity to boundary tokens; and we replace distributional modality-affinity objectives with **counterfactual expert utility** — the measured marginal contribution of an expert under a leave-one-out probe — clustering experts by utility correlation into groups that double as placement units for expert-parallel deployment. Across 3 MoE-VLM backbones and 9 benchmarks, CoDA-MoE improves multimodal accuracy by 1.2% and language-only accuracy by 2.5% over modality-identity routing while reducing expert-parallel communication by 18.6% at matched average sparsity, and at an equal expert-activation count demand-conditioned routing still outperforms fixed top-k by 0.9%, attributing the gain to budget reallocation rather than additional capacity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts, vision-language models, conditional computation, expert routing, expert parallelism, efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ndXnOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoDA-MoE, a demand-conditioned sparse activation framework for MoE-based vision-language models. The key motivation is that a token’s modality identity does not necessarily reflect its actual computational need, particularly its dependence on complementary cross-modal information. To address this issue, the authors introduce a lightweight demand probe that estimates token-level cross-modal demand and dynamically determines the number of experts activated for each token under a fixed global activation budget. Importantly, the demand signal controls only how many experts are activated, while the identities of the selected experts are still determined by the original router. At inference time, the demand estimation is performed in a single backbone pass.

Critical strengths
  1. The paper provides a relatively complete and well-structured formulation of demand-conditioned routing. It clearly separates single-pass demand estimation, integer-valued expert-count allocation under an explicit global sparsity constraint, and expert grouping and placement based on leave-one-out utility. In particular, the allocation problem is explicitly formulated under a fixed activation budget, and the experiments carefully control the average number of expert activations, enabling meaningful comparisons at matched compute budgets.

  2. The paper considers both predictive performance and expert-parallel communication efficiency. The ablation studies cover several major design choices, including the demand probe, demand-conditioned budget allocation, utility-based expert grouping, and placement constraints. This provides useful evidence about the contribution of the individual components rather than reporting only end-to-end accuracy improvements.

Critical weaknesses
  1. The evaluation across model backbones is still limited, making it difficult to assess whether the proposed method generalizes to larger and more recent MoE-VLMs. The current experiments include only DeepSeek-VL2-Tiny, DeepSeek-VL2-Small, and MoE-LLaVA-3.6B×4. While these models cover different expert granularities, the evaluation does not include the full DeepSeek-VL2 model or more recent MoE-VLM architectures. This is particularly relevant because the paper aims to present CoDA-MoE as a general routing framework rather than a model-specific technique. Experiments on stronger and more recent backbones, such as the full DeepSeek-VL2, Kimi-VL-A3B, Qwen3-VL-30B-A3B, or Qwen3.5-35B-A3B, would provide substantially stronger evidence for the scalability and generality of the method.

  2. The related-work discussion omits recent work that is highly relevant to the paper’s central motivation and problem formulation. One particularly important example is AnyExperts [1], which also argues that different tokens in multimodal MoE models require different amounts of computation and proposes on-demand, budget-aware dynamic routing that allocates variable expert computation according to token-level semantic importance. Given the substantial overlap in motivation, the paper should explicitly clarify the differences between the two approaches, particularly in terms of the definition of token demand/importance, the expert-allocation mechanism, and compute-budget control. Another closely related work is Alloc-MoE [2], which explicitly formulates expert activation as a constrained activation-budget allocation problem and redistributes expert activations at both the layer and token levels. Although Alloc-MoE focuses on general MoE inference rather than specifically on multimodal MoEs, its formulation is clearly relevant to the budgeted expert-allocation problem studied here. These works should be discussed to better position the novelty of CoDA-MoE relative to recent budget-aware adaptive routing methods.

[1] Yuting Gao, Lan Wang, Hengyuan Zhao, Linjiang Huang, Si Liu, and Qingpei Guo. AnyExperts: On-Demand Expert Allocation for Multimodal Language Models with Mixture of Experts. CVPR Findings, 2026.

[2] Baihui Liu, Kaiyuan Tian, Wei Wang, Zhaoning Zhang, Linbo Qiao, and Dongsheng Li. Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference. ACL, 2026.

  1. The experimental comparison with existing token-adaptive routing methods is insufficient to establish that cross-modal demand is a more effective allocation signal. The paper discusses several lines of prior work on variable expert activation, including Adaptive Gating, dynamic routing, XMoE, dynamic Top-p routing, AdaMoE, and MoE++. It explicitly acknowledges that variable expert counts and token-adaptive computation have already been explored, while distinguishing CoDA-MoE primarily through its use of cross-modal dependence as the allocation signal. However, none of these representative adaptive-routing methods is included as a direct experimental baseline. The current experiments only demonstrate that CoDA-MoE outperforms fixed Top-K routing and a modality-identity-based allocation baseline at matched activation budgets. Consequently, the current evidence does not establish whether cross-modal demand itself is a better allocation signal than previously proposed confidence-, importance-, or difficulty-based signals.
Questions for authors

See weaknesses above.

Reviewer_4KscOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CoDA-MoE, which treats how many experts a token activates separately from which experts it activates in multimodal MoE models. The original router still ranks the experts; a lightweight probe estimates each token's "cross-modal demand", i.e., how much it depends on the other modality, supervised by the change in hidden states when the other modality is replaced by a null input; and an integer allocator assigns expert counts to tokens under a per-layer, per-step activation budget by the marginal gains of a logarithmic surrogate, with feasibility and optimality proven for this surrogate (Proposition 4.1). In addition, expert utility is estimated from the local loss change when each expert is removed, experts are grouped by the correlation of their utilities, and the groups are placed on devices under communication, memory and load constraints. Experiments on DeepSeek-VL2-Tiny, DeepSeek-VL2-Small and MoE-LLaVA show that, at the same average number of activated experts, CoDA-MoE improves the nine-benchmark multimodal average by about 1.2 points and the language-only average by 2.3–2.9 points over a "modality-identity" allocation baseline, and reduces expert-parallel communication by about 18%.

Critical strengths
  1. The paper cleanly separates how many experts a token activates from which experts it activates. It uses a token's dependence on the other modality, rather than its modality label, to decide the amount of computation, and gives an integer allocator that is feasible under the global budget and optimal for the surrogate objective; the proof of Proposition 4.1 is correct. This idea is a useful reference for dynamic compute allocation in multimodal MoE models.
Critical weaknesses
  1. The method run in the experiments does not match the one described in the method section, and no existing method is compared against.

    • (a) The method section (§4.5) freezes the backbone and fits the probe separately on paired full / complementary-modality-removed inputs; the probe is a low-rank projection plus an MLP with complementary-modality context (Eqs. 4–5), and the allocator has no learnable parameters. The experiments (§5.1, Table 7) instead describe the probe as "a linear head" in a single pass, and train the probe, allocator, gates and attention LoRA jointly for 12k steps.

    • (b) Eq. (3) defines the budget per layer and per routing step and states that capacity is not borrowed across layers, whereas Appendix D and Figure 6 state that early layers receive more than 2.41 experts and later layers fewer, so that "the sparsity contract is global rather than per-layer".

    • (c) The only baselines are Dense, fixed Top-K and modality-identity routing, all defined by the authors; §2.4 states that modality-identity routing does not correspond to the implementation of any existing method, and the paper does not state whether these baselines received the same 12k-step LoRA training. None of the variable-expert-count methods discussed in Section 2 (e.g., Adaptive Gating, top-p dynamic routing, DynMoE, AdaMoE, MoE++) is included in the experiments. Among them, top-p dynamic routing, which sets the number of experts by router confidence, is the closest to this work, while the paper's central claim is precisely that cross-modal demand is a better signal than router confidence for deciding the number of experts (§2.1).

    This determines whether I can attribute the gains to demand-based routing: the improvement over the baseline is only about 1.2 points, and if only CoDA-MoE was trained, or if the allocation actually run differs from the equations, these gains cannot be taken as evidence for the demand signal; without a comparison against existing variable-expert-count methods at the same budget, the paper's central claim also lacks direct evidence.

  2. Several results are inconsistent with the paper's own definitions and need to be verified.

    • (a) Table 9 reports a result for Kmax⁡=2K_{\max} = 2 with Kˉ=2.41\bar{K} = 2.41 fixed, but with at most two experts per token the average cannot reach 2.41.

    • (b) In Table 6, removing utility clustering or the placement constraint changes the multimodal average (65.23→64.94/65.0365.23 \to 64.94 / 65.03). By the paper's definition, placement only changes which device hosts each expert and leaves the token–expert pairs unchanged, so accuracy should be identical.

    • (c) Under Eq. (11), each token sends and receives one copy of its hidden state per remote device. With DeepSeek-VL2-Tiny's hidden size of 1280 in bf16, this bounds the volume at about 12 MB per 1,000 tokens per layer, whereas Table 4 reports 2.74 GB, roughly 200 times larger.

    • (d) The three backbones differ greatly in the number of experts (64 vs. 4) and in scale, yet the per-benchmark gains over the baseline are nearly identical: 1.94 / 1.89 / 1.92 on MMMU, 1.68 / 1.69 / 1.69 on MathVista, 0.84 / 0.84 / 0.90 on TextVQA, and 1.20 / 1.19 / 1.20 on the multimodal average.

    These issues currently prevent me from treating the main tables and ablations as evidence that the method works. Some of them may be typos or results from different experimental versions; I would ask the authors to address them one by one and to provide the raw evaluation logs and variance over multiple runs.

Questions for authors
  1. Which version of the method do Tables 1–6 correspond to: the frozen-backbone version with a separately fitted probe, or the version trained jointly for 12k steps? What is the actual probe architecture, and which parameters of the allocator are trainable? Is the budget independent per layer or shared across layers?

  2. Were the modality-identity and fixed Top-K baselines trained with the same 12k-step LoRA as CoDA-MoE? If not, could you compare them under the same training conditions?

  3. Could you compare against existing variable-expert-count methods at the same average activation count and under the same training conditions, including at least top-p dynamic routing, which sets the number of experts by router confidence? This is the most direct test of the central claim that the demand signal is better than router confidence, and its outcome would substantially affect my assessment.

  4. Please explain the four issues in Critical Weakness 2: how the Kmax⁡=2K_{\max} = 2 result in Table 9 was obtained, why the placement ablation changes accuracy, how the communication volume in Table 4 was measured and in what units, and why the per-benchmark gains are nearly identical across the three backbones. Could you provide the raw evaluation logs and the variance over multiple runs?

  5. Please specify the exact checkpoints of the three backbones and the setting of the "Dense" row in each table. "Dense (all experts)" in Tables 1–2 is actually the original Top-6 activation of DeepSeek-VL2, not all 64 experts, while the MoE-LLaVA "Dense" row in Table 3 activates all 4 experts, whereas the released MoE-LLaVA models all use Top-2, i.e., twice the original setting. The meaning of "Dense" is therefore inconsistent across the three tables. In addition, no released MoE-LLaVA model is named "3.6B×4"; 3.6B is the activated parameter count of MoE-LLaVA-2.7B×4-Top2, so please confirm which model was actually used.

  6. The following minor issues do not affect my overall assessment and are offered for revision.

    • (a) The x-axis of Figure 6 spans about 24 layers, whereas DeepSeek-VL2-Tiny has 12 layers.

    • (b) §6.1 states that "21% of tokens use three or more experts", but the bars in the right panel of Figure 4 sum to about 39%.

    • (c) Table 7 describes the probe as "linear" while also listing a hidden width of 256.

Reviewer_vGfBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes CoDA-MoE -- a demand-conditioned trainable routing framework that estimates the token-level cross-modal demand to adaptively allocate the tokens to variable number of experts instead of standard fixed K. The method is evaluated on 3 MoE VLM backbones across 9 multimodal benchmarks, and improves average multimodal accuracy by 1.2 points, and language only accuracy by 2.5 points while cutting up to 18.6% of expert-parallel communication.

Critical strengths

The paper addresses a relevant problem occurring in MoE models, specifically in VLMs, where different modalities are joint together. The method aims to improve deployment efficiency by adding traffic load awareness. The MoE efficient deployment remains an open problem, and research community would benefit from progress made towards reducing communication overhead and more efficient expert allocation.

Critical weaknesses
  1. The text is poorly written, it is hard to follow. It is not clear what is the main message and the goal of the paper: is it improving accuracy, establish a better compute allocation for model deployment or something else? The main motivation is built on an unclearly chosen baseline (see Q1). I highly recommend the authors to rewrite the paper to provide a better flow, clearer motivation and method description.

  2. Despite Ablation section (5.4), I still find that evidence that the necessity of introducing a trained probe is limited. E.g., Table 5 shows that attention-mass proxy is only slightly inferior to the proposed method (-0.6 MM points, , 4.1% in communication), which actually requires training. This ablation is done only on one setting, and the gain is modest compared to to this untrained baseline (I do acknowledge though that VLM benchmarks are saturated and even numerically small gain can still be an improvement). Only one setting in Table 5 actually ablates the necessity of the training, which is attention-mass proxy.

I find that in the current form, the paper is not reader-friendly, and its motivation and advantage of the proposed method are not fully supported. I highly encourage authors to address these issues in further manuscript versions.

Questions for authors
  1. [clarification request] I have troubles understanding the motivation.

On lines 39-40 in the Introduction, the authors state "... CoDA-MoE investigates whether the latter quantity is a better control signal for distributing a fixed amount of expert computation." And on lines 43-46: "Conversely, a visual token and a text token can both participate in an unresolved alignment, so assigning disjoint budgets solely by origin can miss their common demand." Finally, in Conclusion, there is a sentence on line 453 "We argued that modality identity is the wrong routing quantity for MoE-VLMs and replaced it with cross-modal information demand. "

So it reads like the motivation is to study whether the modality is useful for expert allocation additionally to standard expert assignment mechanism . What I do not really understand is why this is tackled as if it is an existing established baseline approach, while in Related work section on lines 106-110 it is claimed that modality-identity routing is just a controlled comparator, not a method from prior work.

My question is the following: why is modality based routing treated as a main baseline approach if it is not an established method from the literature? And, if it is, could the authors clarify which prior work treats the token modalities as a predictor for activated experts per token instead of standard fixed top-K routing? If there is indeed no relevant prior work, why it is chosen instead of existing adaptive routing baselines, e.g., mentioned by the authors in Section 2.2 AdaMoE and MoE++ ?

  1. [clarification request] Second, not only the probe is being trained, but other parameters are updated too: "Routing components, the probe, the allocator, and LoRA adapters on attention are trained for 12k steps with the expert FFNs and vision encoders frozen; hyperparameters are listed in Appendix A" [line 297]. I did not find the explicit ablation disentangling additional training and introduced probe. Or are the setups in tables, other than CoDA-MoE, also trained in a similar manner with updated routing components and attention?

Additional remarks:

  • I could not locate the required AI USE STATEMENT section. Also, the formatting is not standard (the page title is "ICLR 2027 / CoDA-MoE" instead of "Under review as a conference paper at ICLR 2027").
  • Sentence on line 45 has repetitive "need", difficult to follow.
  • Section 2.4 should be rewritten for better clarity.
  • Figures 2, 3 and 4 have overlapping text.
  • Font in Tables 4, 8 and 9 is too large.
  • I find bold formatting in tables misleading (limen 329: Bold marks the best budget-matched (modality-identity vs. CoDA-MoE) result per column) as it does not indicate the best setting in general. So it should distinguish between the best overall setting or the best setting within matched budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.