- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes OmniMoE-VL, a sparse vision-language model that introduces question-conditioned visual-depth routing to address the limitations of fixed cross-modal interfaces. The method uses a shared global route to coordinate local patch fusion and dynamic visual injection into the language model, alongside token-level MoE routing in the vision and language stacks. Experiments across eight benchmarks show competitive performance, and the ablation studies and route intervention analyses provide evidence for the effectiveness of the proposed design. Overall, the paper presents a reasonable motivation and an interesting architectural approach. My main concerns are related to the experimental evaluation, particularly the sensitivity to key design choices, the lack of qualitative visualizations, and the analysis of training costs.
Critical strengths
-
The paper identifies a relevant limitation of existing MoE-based VLMs, where visual representations are typically fixed before language-side expert routing. The motivation for question-conditioned visual-depth selection is reasonable, as different questions may require information from different levels of visual representations.
-
The proposed method combines global visual-depth routing, local patch fusion, and dynamic visual injection. Sharing the depth preference across fusion and injection is an interesting design choice, as it allows the two pathways to coordinate visual information while preserving local spatial adaptation.
-
The paper includes experiments across eight benchmarks, along with architectural ablations, matched routing comparisons, and route intervention analyses. These experiments cover several aspects of the proposed method, and the results provide some evidence for the effectiveness of visual-depth routing and the benefits of coupling the two pathways.
Critical weaknesses
(1) The paper adopts a four-stage training procedure involving the projector, language experts, vision experts, and joint optimization, using approximately 16.2M multimodal instruction samples. However, the authors do not report the training time, GPU-hours, or total computational cost of each stage. This makes it difficult to assess the actual training overhead of the proposed method and the trade-off between its performance gains and additional computational costs.
(2) The paper involves several hyperparameters, and one design choice I am particularly concerned about is the use of 16 query prototypes combined into 8 router queries. In comparison, BLIP-2 uses 32 learnable queries. Although these queries serve different purposes, the number of queries may still affect visual information extraction, routing quality, and inference cost. However, the paper does not provide a sensitivity analysis of this choice. In addition, the authors adjust the effective number of queries based on prompt length, but longer prompts do not necessarily imply more complex visual understanding or reasoning tasks. These are some of my concerns regarding the experimental evaluation.
(3) For a multimodal learning paper, I find the current qualitative analysis somewhat limited. In particular, a key motivation of the paper is to dynamically select visual depths based on different questions, but there are few intuitive examples illustrating how this mechanism works. For instance, for the same image, which questions rely more on shallow visual features, which require deeper semantic representations, and whether these selections align with the actual visual evidence needed. Providing more visualizations and qualitative examples would help readers better understand the routing behavior and further strengthen the paper's core motivation.
(4) The paper primarily uses a fixed resolution of 384 × 384 and focuses on single-image inputs. Given the growing need for multimodal models to handle high-resolution images and complex visual scenes, this relatively limited experimental setting makes it difficult to fully assess the method's generalization ability and practical applicability. I consider this concern less critical than the issues raised above, as addressing it would require substantial additional experiments. I would therefore encourage the authors to focus on discussing and clarifying the earlier concerns. Overall, I find the motivation and methodology of this paper promising, and addressing these experimental concerns would strengthen the paper's empirical support and increase my confidence in its contributions.
Questions for authors
Please refer to the Critical Weaknesses section.
5 comments
Response to iv8N (2/2): Routing examples and input scope
Real routing examples, same-image comparisons, and interventions connect depth selection to prediction quality. (Weakness 3)
We have added four real-routing case studies in Appendix F of the revised supplement: bottle labels, a camera, a chalkboard, and a fuel-price sign. Each image has four questions. Figures 4–7 show PCA features from all ten candidate encoder depths, Figures 8–11 show the image, question, and the four depths actually selected by the global route for each image–question pair. Panels (b)–(e) visualize the selected layers, with annotated evidence regions linking their spatial features to the question.
The examples make the question-dependent changes concrete:
| Case | Question contrast | Actual selected encoder blocks |
|---|---|---|
| Bottle labels (Fig. 8) | Read the brand → read the age statement | {1,9,21,29} → {1,5,13,33} |
| Camera (Fig. 9) | Read the brand → identify the object | {1,5,9,25} → {1,13,21,37} |
| Chalkboard (Fig. 10) | Read the middle expression → compare the two outer answers | {1,17,21,33} → {5,9,13,21} |
| Fuel prices (Fig. 11) | Read the Plus price → identify the grade above Diesel | {5,9,17,29} → {1,29,33,37} |
The image is fixed within each case, while the question changes both the required evidence and the selected depth set. The annotations identify the relevant label regions, object extent, written expressions, or neighboring rows, making the connection between visual evidence and routing choices directly inspectable.
Table 5 quantifies this behavior across images. Each comparison fixes the image and target question, then compares its route with a paraphrase and another question about that image:
| Comparison | Images | Targets | Different-question | Paraphrase | |
|---|---|---|---|---|---|
| Fixed target | 283 | 322 | 0.301 | 0.162 | 0.139 |
| Matched structural type | 136 | 174 | 0.282 | 0.159 | 0.123 |
| Matched answer length | 193 | 214 | 0.264 | 0.171 | 0.093 |
| Identical reference answer | 44 | 53 | 0.217 | 0.145 | 0.072 |
Here is total variation. Eligible comparisons are averaged within targets, then within images, with equal weight per image. The latter three rows are separate matched subsets. The positive 0.072 gap persists when different questions have exactly the same reference answer, linking route variation to the question even when its output is fixed.
Table 5 further tests the effect on prediction. Holding the target image, question, query features, and dynamic operator factors fixed, transferring another question's route increases reference-answer NLL by 0.04 through fusion, 0.03 through injection, and 0.09 through both, versus 0.01 for a paraphrase route, in nats per answer token. The four case studies show which depths change with the question, the paired comparisons establish the broader pattern, and the interventions connect question-matched routing to prediction quality through both visual pathways.
Preserving the spatial patch grid provides a direct path to higher-resolution, multi-image, and video inputs. (Weakness 4)
At the evaluated resolution of 384 × 384, OmniMoE-VL retains all 576 patch positions while selecting among visual depths. Depth routing and spatial resolution therefore serve complementary roles: routing determines which representations contribute at each position, while the patch grid preserves the spatial organization of the image. This design maintains access to local visual details relevant to text recognition and document understanding.
The same interface can be applied to higher-resolution inputs by operating on the resulting larger patch grid. For multi-image and video inputs, prompt-conditioned depth routing can operate on each image or frame, preserving its patch layout and reusing its depth preference for fusion and language-layer injection. Image identifiers and temporal position embeddings can distinguish visual sources, while the shared prompt guides depth selection across them.
These extensions preserve the central architectural principle: adapting visual access to the question while retaining spatial structure. Our single-image experiments provide the empirical basis for this design, and higher-resolution and multi-image/video studies would evaluate it under larger visual-token budgets and additional cross-image or temporal dependencies.
Thank you for your detailed responses and clarifications. My main concerns have now been largely addressed. After also considering the comments and discussions from other reviewers, I am happy to increase my score. I appreciate your efforts throughout the rebuttal process and wish you the best of luck!
Response to iv8N (1/2): Training cost and query design
Thank you for the constructive assessment and for identifying training cost, query capacity, and qualitative routing behavior as the main points to strengthen. We address each below and distinguish the query bank's routing role from the visual-token interface.
Stage-wise training time and computational cost analysis. (Weakness 1)
Four-stage training cost. We thank the reviewer for highlighting the need for a more detailed analysis of the training cost. To address this concern, we provide a stage-wise breakdown of the computational cost for OMVL-16M, covering all four training stages, including projector training, language expert training, vision expert training, and joint optimization. The GPU-hours consumed by each stage are reported below.
The complete four-stage training of OMVL-16M was conducted using 16 NVIDIA H800 GPUs, consuming a total of 4,714.80 GPU-hours, equivalent to approximately 12.28 days of training. As shown in the table, Stage II accounts for the largest share of the computational cost, followed by Stage IV and Stage III, while Stage I requires substantially fewer computational resources. These detailed measurements clarify the computational overhead associated with each training stage and provide a concrete reference for evaluating the training efficiency of our method.
Sixteen prototypes and eight router queries provide a compact routing representation while preserving the full visual grid. (Weakness 2)
The sixteen latent seeds are initialized from N(0,0.022) and learned jointly with the projector. They cross-attend to the valid prompt embeddings to produce sixteen question-conditioned prototypes. Eight learned mixtures then combine these prototypes into routing descriptors, which score the ten visual-depth streams. The prototype bank therefore represents the question, while the descriptors organize that representation for depth selection.
The resulting global depth preference guides group-local fusion, where local image–question compatibility refines how features are combined at corresponding spatial positions. The output retains the complete 24×24=576 patch grid. The sixteen prototypes and eight descriptors thus determine the capacity of the routing representation while preserving the full spatial visual sequence.
This clarifies the comparison with BLIP-2: its queries form a compact visual interface, whereas our queries support depth selection. We fix the prototype and descriptor counts across the relevant controlled comparisons to maintain a consistent query capacity and routing budget. This allows us to examine how query participation, visual-depth selection, and coupled injection affect performance without confounding their effects with changes in configuration. Our experimental focus is on understanding these architectural mechanisms rather than optimizing each count independently.
Prompt length regularizes soft query participation within a fixed bank. (Weakness 2)
The gate uses
\qquad n_{\mathrm{tgt}}(L_T)=\operatorname{clip}\!\left(\log(1+L_T)+4,\,2,\,16\right).$$ This target regularizes the concentration of query weights within a fixed bank. It is soft, logarithmic, and clipped, it neither assigns a depth from prompt length nor changes the number of executed query vectors. The actual prototypes and depth scores depend on prompt content. A longer prompt therefore supplies a larger participation target without being declared a harder visual task. The relevant trained control is Table 3(b): replacing the query-participation design with a softmax gate, while retaining the query synthesizer, sixteen prototypes, ten depths, top-4, and auxiliary regularization, changes the score from 83.7 to 81.8 under the same validation/tuning budget. |Query-gating variant|Avg.| |---|---| |Full query-participation design|**83.7**| |Plain softmax gate + matched auxiliary regularization|81.8| |Improvement|**+1.9**| This evaluates the complete gate design. It does not isolate length adaptation alone. Likewise, Table 5's answer-length control concerns the reference answer and should not be interpreted as a prompt-length ablation.