acceptodds
Under review as a conference paper at ICLR 2027

IP-MoE: Investigating Bilinear Vision–Language Interactions for Expert Weighting

Abstract

Mixture-of-Experts (MoE) architectures provide a mechanism for combining expert computation through input-dependent weighting. In vision–language models, however, it remains unclear whether explicit interactions between visual and textual features provide useful information for determining these weights. We investigate this question with IP-MoE, a soft dense mixture of experts conditioned on token–patch interactions. IP-MoE constructs a spatial interaction map from question-token and image-patch representations while preserving the image layout, and uses it to assign soft weights to global, local, and hybrid experts. Their weighted outputs, together with image and interaction features, are projected into prefix tokens for a causal language model. We evaluate IP-MoE on seven benchmarks spanning visual question answering, spatial perception, object hallucination, general vision–language abilities, and vision-indispensable reasoning. We further compare bilinear and sum-based interactions, as well as learned and uniform expert weighting. In the evaluated configuration, the model trained with learned expert weighting obtains higher scores on 2D spatial perception and object hallucination, but lower scores on 3D spatial perception, than the model trained with uniform weights. Bilinear interactions likewise yield task-dependent differences relative to sum-based interactions. Training diagnostics indicate nearly uniform gates late in training, and restricting inference to individual expert families produces modest changes in aggregate scores. These findings characterize the behavior and limitations of interaction-conditioned expert weighting, without establishing consistent performance gains or functional expert specialization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision--Language Models, Mixture of Experts, Bilinear Pooling, Visual Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_shZEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether explicit image–question interactions can help a multimodal model assign useful weights to different experts. IP-MoE constructs a spatial interaction map from image patches and question tokens, processes it with six global, local, and hybrid experts, and passes the combined expert features to an LLM together with image and interaction prefixes. The main experiments compare bilinear interactions with learned or uniform expert weights, and additive interactions with learned weights.

Critical strengths
  1. The analysis goes beyond aggregate benchmark scores and identifies behaviors that matter for interpreting the results. In particular, the late-training gate statistics and the CV-Bench-3D answer distribution expose problems that would be easy to miss from accuracy alone. These observations are useful for understanding what a seemingly reasonable multimodal expert design actually learns.

  2. The statistical analysis is careful about what the available evidence can establish. The paired, grouped bootstrap respects the evaluation structure, and the paper distinguishes uncertainty over evaluation examples from variability across training runs. It also avoids treating a confidence interval containing zero as evidence of equivalence. This makes the reported comparisons more credible, even though the experimental scope remains limited.

Critical weaknesses
  1. The technical novelty is incremental. Low-rank bilinear fusion, token–region interactions, and attention/convolution-based processing are established ideas. I see the main addition here as using the same spatial interaction map for expert processing, routing, and a direct language-model prefix. This is a reasonable design, but the paper does not establish a clear advantage or capability arising from this particular combination. The three configurations have mixed rankings, and there is no matched simpler connector that would show whether the added structure is worthwhile. As a result, the architecture alone provides a relatively modest contribution, and much of the case for the paper rests on its analysis.

  2. The analysis does not yet deliver a sufficiently clear finding beyond the behavior of these particular runs. Nearly uniform gates and small differences between expert-family evaluations are interesting observations, but they leave several explanations open: the experts may provide similar useful features, the decoder may rely mainly on the direct prefixes, or the training procedure may favor uniform weights. The paper discusses these possibilities without distinguishing them. The gate measurements are also taken from training microbatches, while the expert-family comparisons use different code versions. I therefore find it difficult to extract a reliable lesson about when interaction-conditioned routing helps, when it fails, or why. The current study documents a concrete system in detail, but the broader insight remains limited.

Questions for authors
  1. What evidence supports complementary roles for the global, local, and hybrid experts? Please distinguish the intended advantage of preserving spatial interaction structure from the more general benefit of conditioning expert weights on both image and text, which is already explored in prior work such as MoVA.

  2. For the same B-L checkpoint, what happens when learned gates are replaced by uniform weights, calibration-set mean weights, or gates shuffled across examples? Keeping the expert outputs and other feature paths unchanged would help establish whether input-dependent weighting contributes to the predictions.

  3. When the A/B options are swapped and predictions are mapped back to the original meanings, does the model preserve its semantic choices or continue selecting A? This result would materially affect whether I interpret the reported score as evidence about visual reasoning.

Reviewer_rkTQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes IP-MoE, a connector architecture for Vision-Language Models (VLMs). It first encodes the image and the query using a frozen CLIP encoder, computes low-rank Hadamard bilinear interactions between every query token and every image patch, and applies softmax-weighted aggregation over the query tokens at each patch to produce a 24×2424 \times 24 query-conditioned spatial interaction map. This map is then fed into 6 experts, comprising 2 global, 2 local, and 2 hybrid experts, whose outputs are combined via a sample-level dense soft router. Finally, the raw image features, the interaction map, and the mixed expert outputs are each projected into 576 tokens, yielding a total of 1,728 visual prefix tokens that are fed into a LoRA-finetuned Llama-3.2-3B.

The authors compare three configurations, each trained only once with random seed 42, and evaluate them across 7 benchmarks: B-L (bilinear interaction with learned routing weights), B-U (bilinear interaction with uniform weights), and S-L (additive interaction with learned routing weights). There are four main empirical observations: (1) there is no consistent performance ranking among the three configurations; (2) by the late stages of training, the router gate entropy reaches 99.56% of its theoretical maximum, indicating nearly uniform routing weights; (3) retaining only a single expert family during inference results in minimal score changes; and (4) on CV-Bench-3D, the B-L variant outputs option "A" for 96.9% of the queries. The authors explicitly acknowledge that these results establish neither performance gains nor expert specialization.

Critical strengths

Candid reporting of limitations: The authors proactively disclose confounding factors that could otherwise make the results appear stronger than they are, such as severe option bias on CV-Bench-3D, opposing trends between F1 and precision on POPE, and conflicting performance directions across sub-tasks in CV-Bench-2D.

Careful statistical treatment: For the four benchmarks where grouping metadata can be recovered, the authors perform group-paired bootstrap resampling and explicitly clarify that these intervals capture only evaluation-sample uncertainty rather than training-run stochasticity.

Critical weaknesses

Limited novelty and lack of positioning against the most relevant literature. Every core component of the proposed architecture has direct precedents that the paper fails to cite: (1) using MoE inside the VLM connector was already explored in CuMo (NeurIPS 2024); (2) text-driven routing in an MoE connector was proposed in Q-MoE (ACM MM 2024); (3) instruction-modulated visual features were introduced in QA-ViT (CVPR 2024) and MoDA (ICML 2026), with the latter adopting a very similar design and achieving positive gains; (4) the interaction operator itself is simply MLB (ICLR 2017); and (5) comparing local convolutional versus global attention projectors was already studied in Honeybee (CVPR 2024). Moreover, the paper's main empirical findings largely echo established conclusions: MM1 (ECCV 2024) already demonstrated that connector architecture matters far less than visual token count and image resolution, while prior MoE routing studies including THOR (ICLR 2022) and Hash Layers (NeurIPS 2021) have shown that learned routing often fails to outperform random or fixed routing and that sample-level routing exhibits weak specialization, which directly applies here, as the proposed router relies on sample-level global pooling. Consequently, as a methods paper, it lacks performance gains; as an analysis paper, it offers no new insights beyond what is already established in the literature.

Missing fundamental baselines and controls. First, the paper lacks a comparison against a standard 576-token MLP connector under the exact same LLM backbone, training data, and compute budget. Second, it omits post-hoc gate interventions on a fixed checkpoint, such as replacing the learned gate with uniform, calibration-mean, or shuffled routing weights. Although the authors list the latter as future work in Appendix I.4, these inference-only interventions require no retraining and incur negligible cost. As it stands, neither of the two main comparisons supports clean causal attribution: B-L vs. B-U compares two independently trained models, whereas B-L vs. S-L simultaneously alters the inputs to the router, the experts, and the prefix without matching feature magnitudes.

Weak empirical performance at a steep computational cost. The model achieves only 62.71 on MMBench (trailing InstructBLIP at 66.75 and BLIP-2 at 65.21), 38.2 on MMStar, and near-random accuracy on CV-Bench-3D, where 96.9% of B-L's predictions are option "A". Meanwhile, feeding 1,728 visual prefix tokens incurs 11.9 TFLOPs, roughly 11×11\times the compute of a comparably sized SpatialLadder model (1.04 TFLOPs).

Minor issues:

  1. Queries are encoded using the CLIP text encoder by splitting text into independent 75-token chunks, which is ill-suited for long prompts containing multiple-choice options.
  2. The main text is overly hedged with disclaimers in nearly every sentence while relegating key empirical evidence to the appendix, which significantly hurts readability.
Questions for authors

Q1. Under the exact same experimental setup, what are the scores of a standard MLP connector across these 7 benchmarks?

Q2. On the trained B-L checkpoint, how much do the benchmark scores change when the learned gate is replaced post-hoc with uniform, shuffled, or dataset-mean routing weights?

Q3. While holding the total prefix sequence length constant, does the router gate still collapse toward a uniform distribution if the two direct bypass prefixes are removed?

Q4. When trained across 3 or more random seeds, do the observed performance differences between B-L and B-U remain consistent?

Q5. What is the fundamental difference between the proposed method and prior works such as Q-MoE and MoDA, and why do MoDA-style approaches achieve positive gains whereas IP-MoE does not? Please also provide a comparison against a CuMo-style MoE-MLP connector under the same LLM backbone and training data.

Reviewer_hovfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether explicit interactions between visual and textual features can provide useful information for expert routing in Mixture-of-Experts (MoE) architectures and proposes IP-MoE. IP-MoE constructs a cross-modal interaction map that preserves the spatial layout of the image through bilinear interactions and patchwise token aggregation. The resulting map is processed by global, local, and hybrid experts, while a router predicts example-level expert weights to combine their outputs. Finally, the original visual features, interaction features, and weighted expert outputs are projected into visual prefix tokens and fed into a language model for answer generation. The paper evaluates multiple configurations across several benchmarks and analyzes the experimental results.

Critical strengths

The paper evaluates multiple configurations across several benchmarks and provides detailed analyses of how different design choices perform across various tasks and settings. The experimental configurations and fine-grained performance results are clearly reported.

Critical weaknesses

1.The evaluation results across multiple benchmarks do not demonstrate clear advantages of the proposed method. Compared with reference models such as SpatialLadder, IP-MoE appears to offer no clear benefits in either performance or computational cost. As a result, the work reads more like an empirical analysis than a convincing methodological contribution. 2.The methodological novelty is limited, and several design choices are insufficiently justified. The main components of IP-MoE appear to be a combination of techniques from existing research. The contribution lie primarily in combining these established components, without clearly demonstrating the benefits of their integration. 3.There is a disconnect between the central research question and the conclusions. The paper initially investigates whether fine-grained vision-language interactions can provide useful information for determining expert weights for different inputs. However, the experiments mainly report performance differences and training observations across different configurations, leaving the central research question insufficiently answered.

Questions for authors

1.Could you further clarify the substantive methodological novelty of IP-MoE compared with existing approaches? 2.Could you provide more direct experimental evidence that IP-MoE enables the model to learn more appropriate expert weights for different inputs, and that these learned weights make a meaningful contribution to the final predictions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.