- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CoDA-MoE, a demand-conditioned sparse activation framework for MoE-based vision-language models. The key motivation is that a token’s modality identity does not necessarily reflect its actual computational need, particularly its dependence on complementary cross-modal information. To address this issue, the authors introduce a lightweight demand probe that estimates token-level cross-modal demand and dynamically determines the number of experts activated for each token under a fixed global activation budget. Importantly, the demand signal controls only how many experts are activated, while the identities of the selected experts are still determined by the original router. At inference time, the demand estimation is performed in a single backbone pass.
Critical strengths
-
The paper provides a relatively complete and well-structured formulation of demand-conditioned routing. It clearly separates single-pass demand estimation, integer-valued expert-count allocation under an explicit global sparsity constraint, and expert grouping and placement based on leave-one-out utility. In particular, the allocation problem is explicitly formulated under a fixed activation budget, and the experiments carefully control the average number of expert activations, enabling meaningful comparisons at matched compute budgets.
-
The paper considers both predictive performance and expert-parallel communication efficiency. The ablation studies cover several major design choices, including the demand probe, demand-conditioned budget allocation, utility-based expert grouping, and placement constraints. This provides useful evidence about the contribution of the individual components rather than reporting only end-to-end accuracy improvements.
Critical weaknesses
-
The evaluation across model backbones is still limited, making it difficult to assess whether the proposed method generalizes to larger and more recent MoE-VLMs. The current experiments include only DeepSeek-VL2-Tiny, DeepSeek-VL2-Small, and MoE-LLaVA-3.6B×4. While these models cover different expert granularities, the evaluation does not include the full DeepSeek-VL2 model or more recent MoE-VLM architectures. This is particularly relevant because the paper aims to present CoDA-MoE as a general routing framework rather than a model-specific technique. Experiments on stronger and more recent backbones, such as the full DeepSeek-VL2, Kimi-VL-A3B, Qwen3-VL-30B-A3B, or Qwen3.5-35B-A3B, would provide substantially stronger evidence for the scalability and generality of the method.
-
The related-work discussion omits recent work that is highly relevant to the paper’s central motivation and problem formulation. One particularly important example is AnyExperts [1], which also argues that different tokens in multimodal MoE models require different amounts of computation and proposes on-demand, budget-aware dynamic routing that allocates variable expert computation according to token-level semantic importance. Given the substantial overlap in motivation, the paper should explicitly clarify the differences between the two approaches, particularly in terms of the definition of token demand/importance, the expert-allocation mechanism, and compute-budget control. Another closely related work is Alloc-MoE [2], which explicitly formulates expert activation as a constrained activation-budget allocation problem and redistributes expert activations at both the layer and token levels. Although Alloc-MoE focuses on general MoE inference rather than specifically on multimodal MoEs, its formulation is clearly relevant to the budgeted expert-allocation problem studied here. These works should be discussed to better position the novelty of CoDA-MoE relative to recent budget-aware adaptive routing methods.
[1] Yuting Gao, Lan Wang, Hengyuan Zhao, Linjiang Huang, Si Liu, and Qingpei Guo. AnyExperts: On-Demand Expert Allocation for Multimodal Language Models with Mixture of Experts. CVPR Findings, 2026.
[2] Baihui Liu, Kaiyuan Tian, Wei Wang, Zhaoning Zhang, Linbo Qiao, and Dongsheng Li. Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference. ACL, 2026.
- The experimental comparison with existing token-adaptive routing methods is insufficient to establish that cross-modal demand is a more effective allocation signal. The paper discusses several lines of prior work on variable expert activation, including Adaptive Gating, dynamic routing, XMoE, dynamic Top-p routing, AdaMoE, and MoE++. It explicitly acknowledges that variable expert counts and token-adaptive computation have already been explored, while distinguishing CoDA-MoE primarily through its use of cross-modal dependence as the allocation signal. However, none of these representative adaptive-routing methods is included as a direct experimental baseline. The current experiments only demonstrate that CoDA-MoE outperforms fixed Top-K routing and a modality-identity-based allocation baseline at matched activation budgets. Consequently, the current evidence does not establish whether cross-modal demand itself is a better allocation signal than previously proposed confidence-, importance-, or difficulty-based signals.
Questions for authors
See weaknesses above.