acceptodds
Under review as a conference paper at ICLR 2027

OmniMoE-VL: A Sparse Vision-Language Model with Coupled Visual-Depth Routing

Abstract

Vision-language models (VLMs) increasingly use sparse mixture-of-experts (MoE) to scale language-side computation, yet visual information is typically routed only after passing through a fixed cross-modal interface. This leaves an important decision unresolved: which intermediate visual representations should be exposed to language computation for a given question? We introduce OmniMoE-VL, a sparse VLM with a coupled visual-depth routed projector. For each image–prompt pair, the projector selects a sparse set of intermediate visual depths and reuses the resulting global preference to guide both local patch fusion and dynamic visual injection into the language model. This design enables question-dependent visual access while preserving the native visual-token sequence, and complements token-level expert routing in the vision and language stacks. Across eight image-based benchmarks, OmniMoE-VL achieves an average score of 85.9 with 28B total and 9B activated parameters. Controlled comparisons show that the routed visual interface provides the dominant architectural gain, while matched route and component controls, same-image route analysis, and route interventions further support the value of coupling and question-conditioned visual access.

Primary area:
foundation or frontier models, including LLMs
Keywords:
VLM, MoE, projector

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_iv8NOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes OmniMoE-VL, a sparse vision-language model that introduces question-conditioned visual-depth routing to address the limitations of fixed cross-modal interfaces. The method uses a shared global route to coordinate local patch fusion and dynamic visual injection into the language model, alongside token-level MoE routing in the vision and language stacks. Experiments across eight benchmarks show competitive performance, and the ablation studies and route intervention analyses provide evidence for the effectiveness of the proposed design. Overall, the paper presents a reasonable motivation and an interesting architectural approach. My main concerns are related to the experimental evaluation, particularly the sensitivity to key design choices, the lack of qualitative visualizations, and the analysis of training costs.

Critical strengths
  1. The paper identifies a relevant limitation of existing MoE-based VLMs, where visual representations are typically fixed before language-side expert routing. The motivation for question-conditioned visual-depth selection is reasonable, as different questions may require information from different levels of visual representations.

  2. The proposed method combines global visual-depth routing, local patch fusion, and dynamic visual injection. Sharing the depth preference across fusion and injection is an interesting design choice, as it allows the two pathways to coordinate visual information while preserving local spatial adaptation.

  3. The paper includes experiments across eight benchmarks, along with architectural ablations, matched routing comparisons, and route intervention analyses. These experiments cover several aspects of the proposed method, and the results provide some evidence for the effectiveness of visual-depth routing and the benefits of coupling the two pathways.

Critical weaknesses

(1) The paper adopts a four-stage training procedure involving the projector, language experts, vision experts, and joint optimization, using approximately 16.2M multimodal instruction samples. However, the authors do not report the training time, GPU-hours, or total computational cost of each stage. This makes it difficult to assess the actual training overhead of the proposed method and the trade-off between its performance gains and additional computational costs.

(2) The paper involves several hyperparameters, and one design choice I am particularly concerned about is the use of 16 query prototypes combined into 8 router queries. In comparison, BLIP-2 uses 32 learnable queries. Although these queries serve different purposes, the number of queries may still affect visual information extraction, routing quality, and inference cost. However, the paper does not provide a sensitivity analysis of this choice. In addition, the authors adjust the effective number of queries based on prompt length, but longer prompts do not necessarily imply more complex visual understanding or reasoning tasks. These are some of my concerns regarding the experimental evaluation.

(3) For a multimodal learning paper, I find the current qualitative analysis somewhat limited. In particular, a key motivation of the paper is to dynamically select visual depths based on different questions, but there are few intuitive examples illustrating how this mechanism works. For instance, for the same image, which questions rely more on shallow visual features, which require deeper semantic representations, and whether these selections align with the actual visual evidence needed. Providing more visualizations and qualitative examples would help readers better understand the routing behavior and further strengthen the paper's core motivation.

(4) The paper primarily uses a fixed resolution of 384 × 384 and focuses on single-image inputs. Given the growing need for multimodal models to handle high-resolution images and complex visual scenes, this relatively limited experimental setting makes it difficult to fully assess the method's generalization ability and practical applicability. I consider this concern less critical than the issues raised above, as addressing it would require substantial additional experiments. I would therefore encourage the authors to focus on discussing and clarifying the earlier concerns. Overall, I find the motivation and methodology of this paper promising, and addressing these experimental concerns would strengthen the paper's empirical support and increase my confidence in its contributions.

Questions for authors

Please refer to the Critical Weaknesses section.

5 comments
AuthorsOpenReview ↗

Response to iv8N (1/2): Training cost and query design

Thank you for the constructive assessment and for identifying training cost, query capacity, and qualitative routing behavior as the main points to strengthen. We address each below and distinguish the query bank's routing role from the visual-token interface.

Stage-wise training time and computational cost analysis. (Weakness 1)

Four-stage training cost. We thank the reviewer for highlighting the need for a more detailed analysis of the training cost. To address this concern, we provide a stage-wise breakdown of the computational cost for OMVL-16M, covering all four training stages, including projector training, language expert training, vision expert training, and joint optimization. The GPU-hours consumed by each stage are reported below.

OMVL-16M stageGPU-hours
I226.29
II1,761.48
III1,148.22
IV1,578.81
Total4,714.80

The complete four-stage training of OMVL-16M was conducted using 16 NVIDIA H800 GPUs, consuming a total of 4,714.80 GPU-hours, equivalent to approximately 12.28 days of training. As shown in the table, Stage II accounts for the largest share of the computational cost, followed by Stage IV and Stage III, while Stage I requires substantially fewer computational resources. These detailed measurements clarify the computational overhead associated with each training stage and provide a concrete reference for evaluating the training efficiency of our method.

Sixteen prototypes and eight router queries provide a compact routing representation while preserving the full visual grid. (Weakness 2)

The sixteen latent seeds are initialized from N(0,0.022)\mathcal N(0,0.02^2) and learned jointly with the projector. They cross-attend to the valid prompt embeddings to produce sixteen question-conditioned prototypes. Eight learned mixtures then combine these prototypes into routing descriptors, which score the ten visual-depth streams. The prototype bank therefore represents the question, while the descriptors organize that representation for depth selection.

The resulting global depth preference guides group-local fusion, where local image–question compatibility refines how features are combined at corresponding spatial positions. The output retains the complete 24×24=57624\times24=576 patch grid. The sixteen prototypes and eight descriptors thus determine the capacity of the routing representation while preserving the full spatial visual sequence.

This clarifies the comparison with BLIP-2: its queries form a compact visual interface, whereas our queries support depth selection. We fix the prototype and descriptor counts across the relevant controlled comparisons to maintain a consistent query capacity and routing budget. This allows us to examine how query participation, visual-depth selection, and coupled injection affect performance without confounding their effects with changes in configuration. Our experimental focus is on understanding these architectural mechanisms rather than optimizing each count independently.

Prompt length regularizes soft query participation within a fixed bank. (Weakness 2)

The gate uses

\qquad n_{\mathrm{tgt}}(L_T)=\operatorname{clip}\!\left(\log(1+L_T)+4,\,2,\,16\right).$$ This target regularizes the concentration of query weights within a fixed bank. It is soft, logarithmic, and clipped, it neither assigns a depth from prompt length nor changes the number of executed query vectors. The actual prototypes and depth scores depend on prompt content. A longer prompt therefore supplies a larger participation target without being declared a harder visual task. The relevant trained control is Table 3(b): replacing the query-participation design with a softmax gate, while retaining the query synthesizer, sixteen prototypes, ten depths, top-4, and auxiliary regularization, changes the score from 83.7 to 81.8 under the same validation/tuning budget. |Query-gating variant|Avg.| |---|---| |Full query-participation design|**83.7**| |Plain softmax gate + matched auxiliary regularization|81.8| |Improvement|**+1.9**| This evaluates the complete gate design. It does not isolate length adaptation alone. Likewise, Table 5's answer-length control concerns the reference answer and should not be interpreted as a prompt-length ablation.
AuthorsOpenReview ↗

Response to iv8N (2/2): Routing examples and input scope

Real routing examples, same-image comparisons, and interventions connect depth selection to prediction quality. (Weakness 3)

We have added four real-routing case studies in Appendix F of the revised supplement: bottle labels, a camera, a chalkboard, and a fuel-price sign. Each image has four questions. Figures 4–7 show PCA features from all ten candidate encoder depths, Figures 8–11 show the image, question, and the four depths actually selected by the global route for each image–question pair. Panels (b)–(e) visualize the selected layers, with annotated evidence regions linking their spatial features to the question.

The examples make the question-dependent changes concrete:

CaseQuestion contrastActual selected encoder blocks
Bottle labels (Fig. 8)Read the brand → read the age statement{1,9,21,29} → {1,5,13,33}
Camera (Fig. 9)Read the brand → identify the object{1,5,9,25} → {1,13,21,37}
Chalkboard (Fig. 10)Read the middle expression → compare the two outer answers{1,17,21,33} → {5,9,13,21}
Fuel prices (Fig. 11)Read the Plus price → identify the grade above Diesel{5,9,17,29} → {1,29,33,37}

The image is fixed within each case, while the question changes both the required evidence and the selected depth set. The annotations identify the relevant label regions, object extent, written expressions, or neighboring rows, making the connection between visual evidence and routing choices directly inspectable.

Table 5 quantifies this behavior across images. Each comparison fixes the image and target question, then compares its route with a paraphrase and another question about that image:

ComparisonImagesTargetsDifferent-question DDParaphrase DDΔD\Delta D
Fixed target2833220.3010.1620.139
Matched structural type1361740.2820.1590.123
Matched answer length1932140.2640.1710.093
Identical reference answer44530.2170.1450.072

Here D(qi,qj)=12∥w(qi)−w(qj)∥1D(q_i,q_j)=\tfrac12\|w(q_i)-w(q_j)\|_1 is total variation. Eligible comparisons are averaged within targets, then within images, with equal weight per image. The latter three rows are separate matched subsets. The positive 0.072 gap persists when different questions have exactly the same reference answer, linking route variation to the question even when its output is fixed.

Table 5 further tests the effect on prediction. Holding the target image, question, query features, and dynamic operator factors fixed, transferring another question's route increases reference-answer NLL by 0.04 through fusion, 0.03 through injection, and 0.09 through both, versus 0.01 for a paraphrase route, in nats per answer token. The four case studies show which depths change with the question, the paired comparisons establish the broader pattern, and the interventions connect question-matched routing to prediction quality through both visual pathways.

Preserving the spatial patch grid provides a direct path to higher-resolution, multi-image, and video inputs. (Weakness 4)

At the evaluated resolution of 384 × 384, OmniMoE-VL retains all 576 patch positions while selecting among visual depths. Depth routing and spatial resolution therefore serve complementary roles: routing determines which representations contribute at each position, while the patch grid preserves the spatial organization of the image. This design maintains access to local visual details relevant to text recognition and document understanding.

The same interface can be applied to higher-resolution inputs by operating on the resulting larger patch grid. For multi-image and video inputs, prompt-conditioned depth routing can operate on each image or frame, preserving its patch layout and reusing its depth preference for fusion and language-layer injection. Image identifiers and temporal position embeddings can distinguish visual sources, while the shared prompt guides depth selection across them.

These extensions preserve the central architectural principle: adapting visual access to the question while retaining spatial structure. Our single-image experiments provide the empirical basis for this design, and higher-resolution and multi-image/video studies would evaluate it under larger visual-token budgets and additional cross-image or temporal dependencies.

Thank you for your detailed responses and clarifications. My main concerns have now been largely addressed. After also considering the comments and discussions from other reviewers, I am happy to increase my score. I appreciate your efforts throughout the rebuttal process and wish you the best of luck!

Reviewer_KKFwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores MoE vision-language models and investigates the routing and fusion of visual input throughout the LLM layers 1-10. In particular, the authors ask: which intermediate visual representations should be exposed to language computation for a given question? The authors propose OmniMoE-VL, a sparse VLM (wht SigLIP and Qwen3 backbone) equipped with a new projector. For each image–question pair, the projector picks 4 of 10 intermediate vision-encoder layers. The paper reports results on 8 benchmarks, reporting (most relatively modest) gains compared to existing open-weights models. The most significant gain seems to be coming from the routed projector component. A same-image analysis shows that the route changes with the question.

Critical strengths

The paper explores a valid and well-motivated hypothesis: which intermediate vision-encoder projections should be routed to the language model, conditioned on the question about the image? It hypothesises that different questions about the same image should produce different routes and changing these routes would hurt the prediction. They susccessfully show that with the architecture they propose, this indeed is the case and that the overall prediction quality improves. They also show that a similar pattern happes in Qwen3-VL, where different questions about the same image rely on a different mix of visual layers. This is an indicator that the router idea is the right premise.

Critical weaknesses

The most critical weakness of the paper is the complexity of the proposed architecture and training methodology (auxiliary losses, training phases, etc.) relative to what each component seems to contribute. I suspect that some of this complexity may not be necessary. For example, it is clear that the routed projector helps (Table 4a), but what about replacing each component with simpler alternatives? For example, replace routed injection with per-layer LoRA, which according to Table 3b loses only 0.4 points.

This is also tied to the Qwen3-VL ablation (Table 5): it seems that even without explicit routing, Qwen3-VL relies on different layers to answer different questions. This suggests that the LLM can select implicitly from multi-layer access. This does not necessarily imply that explicit routing is not useful; it is difficult to assess whether the gains of the proposed method can be attributed to the router or to multi-layer access itself, so a static multi-layer baseline would help in strengthening the central contribution of the paper.

Questions for authors

Both my questions refer to suggestions to future work or improvement.

  • It would be useful to have a baseline that fues the same ten layers with fixed learned weights, and a DeepStack-style injection baseline. If static fusion gets close to 85.2, the case for question-conditioned routing gets much weaker.

  • The backbone of the proposed model is Qwen3, so Qwen3-VL is a natural baseline: however Qwen3-VL baseline results are missing for a few of the datasets, which makes this comparison weaker. Any particular reason why? A full comparison would show whether the accuracy gain is worth 4.7× the memory.

3 comments
AuthorsOpenReview ↗

Response to KKFw (1/2): Static access and question-dependent routing

Thank you for recognizing both the question-dependent visual-access hypothesis and the experiments connecting routing to predictions. We agree that the important comparison is with models that already have multi-layer access. We clarify that comparison and distinguish the complete interface from its individual refinements.

Dense Connector and DeepStack provide direct multi-layer baselines. (Weakness 2; Question 1)

Table 4(b)'s “MLP-only projector” implements Dense Connector: its MLP head follows static multi-layer integration. We now provide eight-task results for that baseline and DeepStack:

Model / visual interfaceEight-task Avg.
Dense Connector80.3
DeepStack79.9
OmniMoE-VL (w/o. Vision MoE)85.2
OmniMoE-VL85.9

Before adding vision experts, the routed-interface model improves over Dense Connector by 4.9 points and DeepStack by 5.3 points. These results compare against models that already use multiple visual layers. The original five-task component study provides a separate internal test:

Projector variantQRIFive-task Avg.
Full projector✓✓✓87.6
w/o. Q&R✗✗✓83.1

Removing query control and structured routing costs 4.5 points while retaining the dynamic injection operator. This internal control is distinct from the DeepStack baseline above.

Route sharing and interventions test the functional value of explicit question-matched routing. (Weakness 2)

Your interpretation of Qwen3-VL is correct, a model with multi-depth access can use those depths differently for different questions without our router. That observation supports the premise that the useful mixture depends on the question. It does not, by itself, establish superiority of explicit routing.

The additional evidence comes from OmniMoE-VL. First, Table 3(b) retains multi-depth access and compares shared versus independent routes at matched parameter count: 83.7 versus 82.6. Second, Table 5 replaces only the route while keeping the target image, prompt, queries, and dynamic operator factors fixed. A paraphrase route increases reference-answer NLL by 0.01 nats/token, whereas another question's route increases it by 0.04 in fusion, 0.03 in injection, and 0.09 in both pathways. These controls show that the route's assignment to the current question has a functional consequence and that sharing improves the measured model.

Thus, implicit question dependence motivates the design, and the trained coupling comparison and the within-model intervention assess its implementation.

AuthorsOpenReview ↗

Response to KKFw (2/2): Complexity and the Qwen comparison

Component replacements distinguish the injection increment from the full interface gain. (Weakness 1)

Table 3(b) already tests the proposed simpler alternatives:

Projector variantAvg.
Coupled projector83.7
Independent fusion/injection routes82.6
Regularized softmax gate81.8
Standard router with load-balance/UCB83.1
Rank/parameter-matched per-layer LoRA83.3

The LoRA row retains the surrounding query/routing system and replaces its injection realization. Its 0.4-point difference is therefore the marginal contribution of that realization. The broader interface increment is 4.9 points in Table 4(a), and the matched coupling increment is 1.1 points in Table 3(b). We distinguish these contributions instead of attributing the full gain to the dynamic operator.

We also tested whether the sparse design translates into reduced execution work. Reusing projections and skipping unselected injection paths reduces projector-plus-prefill-injection time from 36.59 to 28.28 ms (22.7%) with bitwise-identical outputs on all eight tested inputs. This is a single-A800 BF16, batch-1.

The inference path also excludes the history prior, underuse bonus, and exploration noise, and these act during early training (Sec. 3.3 and Appendix A.3). At evaluation, the model computes a content-based route, refines fusion locally, and reuses the route for factorized layer updates.

The dense MLP control complements the same-backbone architectural comparisons. (Question 2)

We have trained a fully dense control using Qwen3-8B + SigLIP2-Giant with a MLP projector under OMVL-16M datasets. This model uses the same pretrained backbone pair as OmniMoE-VL and provides a concrete dense reference alongside our sparse architectural variants:

Model / configurationEight-task Avg.
Dense MLP (Qwen3-8B + SigLIP2-Giant)51.4
LLM-only MoE80.3
+ routed projector85.2
Full OmniMoE-VL85.9

The dense control characterizes the backbone pair with a conventional visual interface. The OMVL-16M comparisons then isolate the architectural contributions with the backbone and data fixed: introducing the routed projector improves the average from 80.3 to 85.2, and vision experts further increase it to 85.9. The 4.9-point projector gain establishes the contribution of the question-conditioned visual interface within the controlled setting. Our current Qwen3-VL-8B comparison covers five shared benchmarks. On the shared tasks, OmniMoE-VL averages 84.9 versus 83.3, while requiring more peak VRAM (52 versus 11 GB) and achieving lower decoding throughput (17.52 versus 19.71 tokens/s). To ensure a fair comparison, we report baseline results using the official evaluation metrics and denote any metrics not reported in the official sources with “-”.

Thank you very much for your time and effort in reviewing our submission.

We would like to kindly follow up on our rebuttal, as the rebuttal period is approaching its deadline. We have carefully addressed each of your concerns point by point in our response, and would greatly appreciate it if you could take a moment to review our rebuttal and share any further thoughts or clarifications you may have.

We would be happy to address any remaining concerns or questions.

Thank you again for your time and consideration!

Reviewer_FL5SOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper investigates which intermediate visual representations should be exposed to language computation for a given question and proposes a sparse vision–language model OmniMoE-VL. The approach extends token-level expert routing with question-dependent visual-depth selection and vision fusion. Guiding from the patch-group fusion at the interface while preserving vision tokens information, OmniMoE-VL couples visual-depth selection and language-layer updates with LoRA for comprehensive system construction. Experiments across benchmarks and ablation studies indicate OmniMoE-VL improvement, and results analysis demonstrates the method casual explainability.

Critical strengths
  1. The coupled design provides a flexible mechanism for question-conditioned visual routing. By coordinating patch-group fusion and language-layer updates, it offers a practical approach to integrating multi-depth visual features with DeepStack-style injection.
  2. The experimental results show consistent improvements on benchmarks covering tasks as VQA. The relatively extensive ablations further support the effectiveness of the proposed components.
  3. Results analysis suggests that different questions about the same image induce different visual-depth routing patterns. These findings support the motivation for question-conditioned visual access and provide useful insights for research on the causal interpretability of routing in VLMs.
  4. This paper is well organized and clearly presented. The layout supports a smooth reading experience, and the explanations are generally accessible and sufficiently detailed.
Critical weaknesses
  1. The baseline using Qwen3-8B and SigLIP2 with DeepStack injection should be included to strengthen the experimental comparison. Such a controlled baseline would help distinguish gains from access to multiple visual depths streams information from those attributable specifically to question-dependent vectors. (refer question 2)
  2. Table 6(b) shows that OmniMoE-VL has substantially higher peak VRAM usage and lower decoding throughput than Qwen3-VL-8B. These results indicate the improvement space in deployment efficiency. Comparisons with models operating under similar memory budgets, together with a clearer breakdown of computational costs, would help characterize the performance–resource trade-off analysis for model selection.

Writing Issues:

Statement around line 038, "The interface therefore controls a decision that language-side routing does not," would benefit from additional supporting discussion or references that clarify its scope and underlying assumptions.

Questions for authors
  1. The method includes several fixed architectural settings, including 16 latent seeds, 16 softly weighted query prototypes, eight router queries, top-4 depth selection, a 576-position patch grid, and the rank setting (min⁡(rT,rG)=32\min(r_T,r_G)=32). How were these constant values chosen or derived, whether the choices are supported by empirical analysis or theoretical considerations. Further clarification of how the 16 latent seeds are initialized and learned, and how the local text score in group-local patch fusion is constructed, would also be helpful.
  2. The introduction of U(xT)U(x_T) in Eq. (10) needs a sensitivity analysis and controlled ablations that isolate how much the contribution it plays. This would help distinguish question-dependent changes in routing weights ω\omega from the effective use of the correct visual depth information.
  3. Figure 3 shows relatively distributed expert load in the LLM stack, whereas the vision stack exhibits much greater variation (as Layer 8). What motivates these different routing budgets and balancing constraints (as Table 1, Vision MoE 1/8 and LLM MoE 2/8)? How do they affect expert utilization and task performance in Figure 2(b) selection.
6 comments
AuthorsOpenReview ↗

Response to FL5S (1/2): Multi-depth controls and query design

Thank you for recognizing the coordinated fusion/injection design, the controlled experiments, and the question-conditioned routing evidence. We clarify how the existing controls separate multi-depth access from route selection, then address the implementation and resource questions.

Static multi-layer controls and route interventions separate visual access from route selection. (Weakness 1)

Table 4(b)'s “MLP-only projector” implements Dense Connector: its MLP head follows static multi-layer integration. We now provide eight-task results for that baseline and DeepStack:

Model / visual interfaceEight-task Avg.
Dense Connector80.3
DeepStack79.9
OmniMoE-VL (w/o. Vision MoE)85.2
OmniMoE-VL85.9

Before adding vision experts, the routed-interface model improves over Dense Connector by 4.9 points and DeepStack by 5.3 points. These results compare against models that already use multiple visual layers. The original component study provides a separate test:

Projector variantQRIFive-task Avg.
Full projector✓✓✓87.6
w/o. Q&R✗✗✓83.1

Removing query control and structured routing costs 4.5 points while retaining the dynamic injection operator.

Two further controls hold multi-depth access in place. Table 3(b) compares shared and independent fusion/injection routes at matched parameter count: 83.7 versus 82.6. Table 5 changes only the route while retaining the target image, question, query features, and dynamic operator factors:

Fusion routeInjection routeΔ\DeltaNLL, nats/answer token
NativeNative0.00
ParaphraseParaphrase0.01
Other questionNative0.04
NativeOther question0.03
Other questionOther question0.09

The trained comparison tests sharing, and the intervention tests whether the route appropriate to the target question improves its prediction. Together they provide evidence beyond simply exposing the language model to additional visual streams.

Language-side expert routing operates on the visual representations supplied by the interface. (Writing issue, line 038)

The intended comparison is with an LLM-only MoE whose visual interface is fixed. The precise statement is:

In an LLM-only MoE with a fixed visual interface, expert routing changes how the supplied representation is processed, and it does not add explicit access paths to intermediate encoder streams omitted by that interface.

This statement allows the language model to reweight any visual information already available to it. Indeed, our Qwen3-VL probe studies that implicit behavior. OmniMoE-VL makes a depth preference explicit at the interface and shares it across two visual pathways.

Learned query descriptors guide local fusion while preserving all 576 visual positions. (Question 1)

The sixteen seeds and sixteen prototypes are successive states of the same learned bank, not two separate sets of extracted visual tokens. In the implementation, the latent bank is initialized with independent N(0,0.022)\mathcal N(0,0.02^2) entries and optimized with the projector. Its vectors cross-attend to the valid prompt embeddings, and a residual seed contribution and normalization produce sixteen prompt-conditioned prototypes (Eq. (3)). Eight learned mixtures of those prototypes then provide descriptors for comparing visual depths (Eq. (5)).

All 576 visual positions remain in the language input. Thus, 16/8 controls descriptor capacity and routing work, while (384/16)2=576(384/16)^2=576 is determined by the image grid. Groups of 32 define 18 local fusion regions, and rank 32 bounds each factorized injection and adapter.

For group bb and depth ss, vb,sv_{b,s} is the normalized mean of that group's projected visual features, and tˉ\bar t is the normalized mean of the eight router queries. Equation (8) gives the local weights:

gb=TopKSoftmax4({⟨vb,s,tˉ⟩0.7+log⁡max⁡(ws,10−12)}s=110).g _b = \mathrm{TopKSoftmax} _4 \left( \left\lbrace \frac{\langle v _{b,s}, \bar{t} \rangle}{0.7} + \log \max(w _s, 10^{-12}) \right\rbrace _{s=1}^{10} \right).

Local image-question compatibility therefore refines the shared global preference. The result mixes features at corresponding spatial positions rather than reducing the token count. We use these values to define a consistent capacity and routing budget across the controlled comparisons. This allows us to examine how query participation, visual-depth selection, and coupled injection affect performance without confounding their effects with changes in configuration. Our experimental focus is on understanding these architectural mechanisms rather than optimizing each count independently.

AuthorsOpenReview ↗

Response to FL5S (2/2): Injection, expert budgets, and resources

The intervention changes the route while holding the dynamic operator factors fixed. (Question 2)

Equation (10) constructs

Ts(x)=Bs(xV)diag(βs(x))U(xT)⊤,βs,j=softmaxj(uj⊤bs,jd).T _s(x) = B _s(x _V)\mathrm{diag}(\beta _s(x))U(x _T)^\top, \qquad \beta _{s,j} = \mathrm{softmax} _j \left( \frac{u _j^\top b _{s,j}}{\sqrt{d}} \right).

The second expression is specified in Appendix A.2, Eq. (18). Here βs\beta_s is an input-dependent alignment vector, not a free scalar hyperparameter. The transformation determines what update a stream provides, wsw_s determines how strongly that update participates at its anchor.

The intervention above isolates the latter: UU, BsB_s, and βs\beta_s remain those of the target image-question pair, and only ww is transferred. Therefore, the increased NLL comes from the route substitution, not from replacing the target's query-dependent transformation. Table 3(b) separately compares the full operator with rank/parameter-matched per-layer LoRA, giving 83.7 versus 83.3. This controlled comparison supports the input-conditioned injection design over a static low-rank alternative with matched rank and parameter count.

Vision and language expert budgets govern token-level utilization separately from depth routing. (Question 3)

Table 1 uses top-2 vision experts during vision specialization and top-1 in the final joint stage, and language blocks use top-2 of eight experts. The final budget assigns one expert computation per visual token and two per language hidden state. These are token-level parameter routes, distinct from the sample-level ten-depth projector route and the group-local fusion weights.

This distinction also explains why the bars in Figure 3 and the depth choices in Figure 2(b) should not be read as the same routing variable. At top-1, each visual token contributes to one expert, at top-2, each language token contributes to two, so their utilization plots have different participation budgets. Layer-specific feature distributions can also produce different loads. Figure 3 reports those realized loads, rather than establishing that equal load is optimal at every layer. The projector's historical prior and underuse bonus operate on visual depths during exploration and are disabled at evaluation (Appendix A.3).

Accuracy, full-model latency, and memory composition clarify the deployment trade-off. (Weakness 2)

On the tasks shared with Qwen3-VL-8B in Table 2, OmniMoE-VL averages 84.9 versus 83.3, alongside the deployment costs reported in Table 6(b):

ModelCommon-task Avg.Peak VRAM (GB)Decode (tokens/s)
Qwen3-VL-8B83.31119.71
OmniMoE-VL84.95217.52

This comparison makes the trade-off explicit: OmniMoE-VL improves accuracy on the shared tasks, while Qwen3-VL-8B has a smaller memory footprint and higher decoding throughput. The architectural contribution is separately supported by the same-backbone, same-data comparison in Table 4(a), where adding the routed projector improves the eight-task average from 80.3 to 85.2.

To characterize costs beyond decoding throughput, we added full-model prefill and time-to-first-token measurements. The profile uses OmniMoE-VL without vision MoE, implemented in PyTorch/HF on one A800 80GB. With BF16, batch size 1, a 384×384 image, 576 visual positions, and 128 generated tokens, we obtain:

Text tokensFull prefill (ms)TTFT (ms)Decode (tokens/s)Peak allocated (GiB)
64156.24156.3119.9755.11
256182.51182.5719.9555.15

Latency and throughput are averaged over four image requests per length after warmup, and memory is the maximum across those requests. Prefill includes the vision encoder, projector, and language model with visual injection. TTFT additionally includes selecting and returning the first token. These measurements complement Table 6(b) with a separately specified single-GPU configuration.

The memory breakdown identifies parameter storage as the dominant cost. In this reference, model weights occupy 54.90 GiB, including 40.50 GiB of language-expert weights, compared with 45 MiB for the ten projected visual streams at batch size 1. Sparse activation reduces the expert computation performed per token, while the full expert pool remains resident in memory.

Together, these results quantify the accuracy, first-token latency, decoding throughput, and memory footprint relevant to deployment.

Reviewer_FL5SOpenReview ↗

More analysis about VRAM supplement

The following data, sourced from the Qwen3-VL Report (Table 3), is provided to assist the authors in expanding the appendix:

ModelsMMBenchTextVQAMMMUOCRBenchChartQAAvg
Qwen3-VL-8B-Instruct85.082.969.689.689.683.3
Qwen3-VL-30B-A3B-Instruct86.183.574.290.386.884.2
OmniMoE-VL87.887.970.189.389.684.9
Reviewer_zmBwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OmniMoE-VL, a sparse vision-language model (VLM) that chooses which layers of its vision encoder to use for each image–question pair. The main problem this paper tackles is that different questions need different visual information. For example, reading a label needs fine detail from early encoder layers, while a spatial question needs context from later layers. Prior works on sparse (mixture-of-experts, MoE) VLMs route tokens only inside the language model. The visual features come through a fixed projector and are the same for every question. The paper proposes to pick an encoder layers per question, and use that single choice in two places: the visual tokens fed to the language model, and the visual updates injected into its early layers. One shared choice keeps the two paths consistent. The projector reads features from 10 encoder layers. Question-based queries pick the top 4 layers (a weight vector w). w guides how each patch region mixes those layers, and it scales low-rank updates added to language layers 1–10. The proposed model's vision encoder is SigLIP2-Giant and the language model is Qwen3-8B. Both get MoE blocks, for 28B total and 9B active parameters. Evaluation covers 8 benchmarks (MMBench, MM-Vet, TextVQA, DocVQA, POPE, MMMU, OCRBench, ChartQA). OmniMoE-VL averages 85.9 across 8 benchmarks with 9B active parameters, ahead of Qwen2.5-VL-7B at 81.2; the question-routed projector adds 4.9 points over an LLM-only MoE baseline. The model uses 4.7× the memory of Qwen3-VL-8B and decodes 11% slower.

Critical strengths
  1. The paper is well-motivated, with clear examples and experiments that show question-dependent encoder-layer selection. The individual components, like Global route with local refinement, Question-dependent injection, and Query participation, are interesting.
  2. The evaluation section is detailed; the proposed model is evaluated on a wide range of benchmarks, with memory and latency overhead reported.
Critical weaknesses
  1. The baselines in the evaluation could be improved. The gains can't be separated from the stronger backbone (SigLIP2-Giant, Qwen3-8B) and the data, because there is no same-backbone dense baseline, and the sparse baselines use much smaller models. The closest methods (DeepStack, Dense Connector, instruction-guided layer fusion, CuMo) are not compared under the same recipe.

  2. The ablation gains are small, and without variance across seeds, it is hard to evaluate all contributions.

  3. Expert layers, deep injection, and question-based layer weighting already exist (CuMo, DeepStack, Li et al. 2025). The new part, sharing one layer choice between the input tokens and the injection, adds only 1.1 points, limiting the novelty of this work.

Questions for authors
  1. How does a dense model with the same SigLIP2-Giant and Qwen3-8B, a plain MLP projector, and the same 16M data and recipe perform? Without it, the gains in Table 2 can't be separated from the backbone. For example, the 11.5-point MMMU gain over Qwen2.5-VL may come from Qwen3-8B.

  2. Closest baselines. How does the routed projector compare under the same recipe with DeepStack or the Dense Connector (Yao et al., NeurIPS 2024)? Qwen3-VL already injects features from several encoder layers [Sec. 5.4].

  3. What is the seed-to-seed variance for Tables 3(b), 4, and 5(b)? Many gaps are 0.4–1.1 points. Can you also report the LLM-only baseline measured on all 8 tasks, instead of derived from a 5-task aggregate [Eq. 20]?

  4. Why route at all? Why choose a top-4 subset of streams instead of fusing all 10 with learned or question-based weights, as the Dense Connector or Li et al. do? All 10 streams are computed anyway, so what does the sparse selection gain?

  5. Why a fixed stream-to-layer mapping? Why is encoder stream s always injected into language layer s [Sec. 3.1]? This sends fine detail to layer 1 and context to layer 10. Were other mappings, or injecting into deeper language layers, tested?

  6. How are some of the design parameters chosen? How were the 10 taps (every 4 blocks), top-4, groups of 32 patches, rank 32, and 16 queries chosen? Is there a sensitivity study, especially for top-k and the number of taps? How would these values change for larger models?

  7. How much of the 52 GB and of the 11% slower decoding comes from expert weights, the 10 projected streams, the injection updates, and two-GPU communication [Table 6(b)]? Can you report prefill time and time to first token, since the projector's extra work happens during prefill? I am also curious if you have any thoughts on how to improve runtime and memory overhead

  8. Training overhead: What GPUs, how many, and how many GPU-hours did each stage take [Table 1]? How much extra training time does the routed projector add over an MLP projector, given that all 10 streams are projected and scored? How much is the memory overhead at training time?

  9. Can this be scaled to multi-image or video prompts?

6 comments
AuthorsOpenReview ↗

Response to zmBw (1/3): Attribution, baselines, and novelty

Thank you for the careful assessment and for recognizing the question-dependent visual access, controlled evaluation, and reported deployment costs. Your questions distinguish three issues: the contribution of the visual interface, the value of coupling its two pathways, and the resources required by the complete model. The submission provides different controls, and we clarify those controls and the remaining system questions below.

The routed interface improves performance with the backbone and data fixed. (Weakness 1; Question 1)

Table 4(a) directly addresses backbone and data attribution. Its three variants use the same SigLIP2-Giant and Qwen3-8B backbones, OMVL-16M training data, and progressive training setting (Sec. 5.1). This comparison isolates the visual interface within the same sparse model. We also report the newly trained dense control under OMVL-16M datasets:

VariantEight-task Avg.
Dense MLP(Qwen3-8B+Siglip2-Giant)51.4
LLM-only MoE80.3
+ routed projector85.2
+ vision MoE85.9

With the language experts held fixed, adding the routed projector improves the eight-task average from 80.3 to 85.2 (+4.9). Introducing vision experts then adds a further 0.7 points, reaching 85.9. The dominant architectural gain therefore comes from the routed visual interface and is already present before vision experts are added, with the backbone and training data unchanged. The 4.9-point attribution uses the OMVL-16M rows.

Existing controls test static multi-layer access, deep injection, and route sharing. (Weaknesses 1, 3; Question 2)

The “MLP-only projector” in Table 4(b) is a Dense Connector interface: its MLP head follows static multi-layer integration. We now provide eight-task results for Dense Connector and DeepStack alongside OmniMoE-VL:

Model / visual interfaceEight-task Avg.
Dense Connector80.3
DeepStack79.9
OmniMoE-VL (w/o. Vision MoE)85.2
OmniMoE-VL85.9

Even without vision experts, OmniMoE-VL achieves 85.2, improving over Dense Connector by 4.9 points and DeepStack by 5.3 points. Thus, the advantage over static multi-layer fusion and deep visual injection is already present in the routed-interface model before adding vision-side expert capacity. Vision MoE contributes a further 0.7 points, bringing the full model to 85.9 and its gains over the two baselines to 5.6 and 6.0 points, respectively.

Table 4(b) provides a complementary internal comparison:

Projector variantQuery control (Q)Depth routing (R)Dynamic injection (I)Five-task Avg.
Full OmniMoE-VL projector✓✓✓87.6
w/o. Q&R✗✗✓83.1

Removing query control and structured depth routing reduces the average by 4.5 points while preserving the dynamic injection pathway. This comparison shows that retaining deep visual injection alone does not recover the full projector’s performance, supporting the contribution of question-conditioned depth selection within the injection-equipped model.

Table 3(b) then isolates route sharing. Both variants retain multi-depth access and separate fusion/injection pathways, while parameter count is matched:

Public-recipe projector controlAvg.
Full coupled projector83.7
Independent fusion/injection routes82.6
Plain softmax gate + matched regularization81.8
Standard router + load-balance/UCB83.1
Per-layer LoRA, matched rank/parameters83.3

Sharing the route improves the average from 82.6 to 83.7 (+1.1). This tests the coupling itself, while the +4.9 in Table 4(a) measures the complete interface's contribution across eight tasks. The component replacements further quantify the contributions of the gate, router, and injection implementation.

CuMo and instruction-guided layer fusion establish precedents for expert-based computation and question-conditioned feature integration. OmniMoE-VL connects visual selection at the input with visual updates inside the language model: one question-dependent depth preference guides both spatially local fusion and the associated layer-wise injections while preserving all patch positions. Equation (13) explains how the two pathways train this shared preference and the matched comparison in Table 3(b) measures the benefit of sharing it.

AuthorsOpenReview ↗

Response to zmBw (2/3): Variability, routing, and configuration

Paired interventions complement single-run ablations, and all eight baseline tasks were measured. (Weakness 2; Question 3)

The reported training comparisons are single-run results. We therefore report the 0.4-1.1 point component differences as measured increments. Their interpretation is strengthened by a separate, paired experiment in Table 5: replacing only the route, with the target image, question, queries, and operator factors fixed, increases reference-answer NLL by 0.04 for fusion, 0.03 for injection, and 0.09 for both, versus 0.01 for a paraphrase route. These interventions assess the route's functional role without retraining, while remaining distinct from seed variance.

The LLM-only MoE baseline was evaluated on each of the eight benchmarks using the same evaluation settings as the other variants in Table 4(a). Its individual scores are:

MMBenchMM-VetTextVQADocVQAPOPEMMMUOCRBenchChartQAAvg.
80.872.982.286.184.364.783.288.180.3

The unweighted average of these eight scores is 80.2875, reported as 80.3 in Table 4(a). We provide the individual results here to make the eight-benchmark comparison explicit. Adding the routed projector raises this average to 85.2, a 4.9-point improvement with the backbone and training data held fixed.

Top-4 selects participating depths after all ten streams are scored. (Question 4)

The Dense Connector baseline directly addresses whether static multi-layer fusion is sufficient. Across eight benchmarks, it obtains 80.3, compared with 85.2 for OmniMoE-VL without vision experts:

Visual interfaceEight-task Avg.
Dense Connector: static multi-layer fusion80.3
OmniMoE-VL: question-conditioned routed interface, without Vision MoE85.2

The 4.9-point improvement is already present before introducing vision experts. This comparison supports the complete question-conditioned interface over static multi-layer integration, while Table 5 further shows that substituting another question’s route worsens predictions with the target image, question, and operator factors held fixed.

We also directly compare top-4 and top-10 execution while retaining question conditioning. Both configurations score all ten candidate depths and preserve the 576-position visual sequence. Their module timings are:

Measured operationTop-4Top-10
Prefill injection5.43 ms13.43 ms
Injection per decode step5.26 ms13.01 ms
Projector + prefill injection28.28 ms36.37 ms

Sparse selection saves work in the subsequent injection paths, reducing the joint module time by 22.3%. These measurements use one A800 with BF16 and batch size 1, and Part 3 details the implementation.

Thus, the Dense Connector comparison establishes the accuracy benefit of the routed interface over static multi-layer fusion, and the top-4/top-10 measurements identify its sparse-execution benefit. The former changes the interface design as a whole, and it does not isolate the accuracy effect of changing only the depth budget.

Fixed anchors preserve depth order, while query and group sizes define capacity and granularity. (Questions 5-6)

The mapping is from encoder blocks 1, 5, …, 37 to language layers 1, …, 10, not between equal block numbers. Fixing these destinations makes the global route the single decision about which predefined paths participate. The visual content and weights remain image-question dependent. We use a monotone map to preserve depth order and avoid introducing another destination router.

The constants have separate roles (Table 7, Appendix A.1):

SettingRole
Ten taps, every four encoder blocksCandidate feature streams spanning encoder depth
Top-4Global depth-participation budget
16 prototypes / 8 router queriesPrompt descriptors and their mixtures for scoring streams
Groups of 32 patchesLocal fusion granularity: 18 groups at the evaluated resolution
Ranks 32 / 32Factorized transformation and layer-adapter capacity
576 visual positionsThe actual (384/16)2(384/16)^2 patch grid

The query bank provides compact prompt descriptors for depth selection, while local fusion preserves all 576 visual positions. Fixed layer anchors let the same depth preference determine both feature participation and the corresponding injection paths, avoiding an additional destination router. We likewise keep query counts, group sizes, and ranks fixed to maintain a simple architecture with an explicit computational budget to avoid introducing another factors.

AuthorsOpenReview ↗

Response to zmBw (3/3): System costs, training, and input scope

Expert storage and layer-wise injection account for distinct resource costs. (Question 7)

Full prefill and time to first token. We added a single-A800 profile covering the vision encoder, projector, and language model with visual injection. This uses the OmniMoE-VL (without vision MoE). With BF16, batch size 1, a 384×384 image, 576 visual positions, and 128 generated tokens, we obtain:

Text tokensFull prefill (ms)TTFT (ms)Decode (tokens/s)Peak allocated (GiB)
64156.24156.3119.9755.11

Latency and throughput are averaged over four image requests per length after warmup, and memory is the maximum across those requests. Prefill starts from prepared GPU inputs and ends when the first-token logits are available. TTFT additionally includes token selection and returning its ID to the CPU. Table 6(b) remains the separate two-GPU deployment measurement.

Memory footprint. Parameter storage accounts for 54.90 GiB in this reference, compared with 55.11-55.15 GiB peak allocated memory. Language-expert weights alone occupy 40.50 GiB, whereas ten projected streams occupy 45 MiB at batch size 1. Parameter storage therefore dominates this configuration’s memory footprint. Attention intermediates, temporary buffers, and other runtime allocations contribute separately to peak VRAM.

Prefill and decoding work. The projector constructs visual tokens and input-conditioned injection factors during prefill. The measured module times are:

Measured operationTop-4Top-10
Prefill injection5.43 ms13.43 ms
Injection per decode step5.26 ms13.01 ms
Projector + prefill injection28.28 ms36.37 ms

Top-4 reduces the joint projector + prefill injection time by 22.3% relative to top-10. Measurements use one A800, BF16, batch size 1, 576 visual positions and 64 text positions, with three warmup calls and twelve repetitions for each of eight inputs.

Engineering optimizations improve top-4 execution efficiency. In our implementation, we cache the visual projections computed during candidate scoring and reuse them in subsequent operations. We also dispatch only the four selected injection paths, avoiding computation on the six zero-weight paths. Together, these optimizations reduce projector + prefill injection time by 22.7%, with identical visual tokens, routing weights, and injection outputs on all tested inputs. The component-wise timings are shown below:

OperationBefore (ms)Optimized (ms)Reduction
Projector23.1222.652.0%
Prefill injection13.425.4359.6%
Injection per decode step12.975.2659.4%
Projector + prefill injection36.5928.2822.7%

This comparison uses one A800, BF16, batch size 1, 576 visual and 64 text positions, with three warmup calls and twelve timed repetitions per input across eight inputs. Visual tokens, routes, and all ten injection outputs are bitwise identical before and after optimization on every tested input. All ten candidates are still projected and scored, reuse removes redundant projection work, while sparse dispatch removes zero-weight injection work.

Stage-wise training costs on OMVL-16M. (Question 8)

For OMVL-16M, we completed the four-stage training on 16 H800 GPUs and took 12.28 days.

OMVL-16M stageGPU-hours
I226.29
II1,761.48
III1,148.22
IV1,578.81
Total4,714.80

Training-memory overhead. Training memory includes parameters, gradients, optimizer states, saved activations, and temporary buffers. At batch size 1, the ten BF16 projected streams contain 45 MiB of feature data.

The interface extends naturally to multi-image and video inputs through image- or frame-wise depth routing. (Question 9)

OmniMoE-VL provides a direct architectural path to multi-image and video inputs. The same prompt-conditioned interface can select visual depths for each image or frame, preserve its patch layout, and reuse the resulting preference for fusion and language-layer injection. Image identifiers or temporal position embeddings can distinguish the visual sources, while the shared prompt guides their depth selection. This preserves the central design: adapting visual access to the question across a richer set of visual inputs. Our experiments establish this design for single-image inputs, multi-image and video evaluation is a natural next step.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We sincerely thank the Area Chairs, Senior Program Committee members, and all four reviewers for the time and care devoted to evaluating our work. We especially appreciate the reviewers' recognition of the motivation for question-dependent visual access, the coordinated fusion/injection design, and the evidence from same-image analyses and route interventions. We are also encouraged by the positive assessments of the benchmark evaluation and controlled ablations. The detailed and constructive questions have helped us sharpen the explanation of the model's contribution, make the existing comparisons easier to identify, and improve the discussion of its computational costs.

Our contribution is OmniMoE-VL, a sparse vision-language model whose visual interface computes one image-question-dependent depth preference and uses it to guide both local patch fusion and dynamic updates at language layers. This connects the choice of visual evidence at the input to its subsequent use during language computation while preserving all 576 visual positions. The experiments assess three complementary aspects of this design: the same-backbone, same-data interface comparison improves the eight-task average from 80.3 to 85.2 (Table 4(a)), sharing the route improves the matched public-recipe comparison from 82.6 to 83.7 (Table 3(b)), and route-only interventions show that transferring another question's route into both pathways increases reference-answer NLL by 0.09 nats/token, compared with 0.01 for a paraphrase route (Table 5). Together, these results connect the architectural improvement, the value of coupling, and the functional role of question-matched visual access.

To address the reviewers' questions and strengthen the empirical support for our work, we have supplemented the submission with additional controlled comparisons, further analyses of the routing mechanism, and more detailed computational measurements. Regarding performance attribution, we provide same-backbone comparisons with Dense Connector and DeepStack, report the individual LLM-only MoE scores across all eight benchmarks, and compare OmniMoE-VL with Qwen3-VL-8B on their shared tasks. To clarify the routing design and its effectiveness, we explain the query bank, local fusion, and dynamic injection mechanisms, including the distinction between routing descriptors and visual tokens, prompt-length-dependent query participation, and the variables held fixed in route interventions. We also revisit the controlled same-image analysis and the Qwen3-VL depth probe, and provide qualitative examples illustrating question-dependent depth selection. Regarding computational costs, we report stage-wise training expenses, end-to-end prefill for the variant without vision MoE and time-to-first-token measurements, memory composition, and runtime savings from projection reuse and sparse injection. We further clarify the architectural settings and discuss the scope of the current single-image evaluation. These additional results and clarifications are presented in our individual responses to the four reviewers below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.