- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies whether explicit image–question interactions can help a multimodal model assign useful weights to different experts. IP-MoE constructs a spatial interaction map from image patches and question tokens, processes it with six global, local, and hybrid experts, and passes the combined expert features to an LLM together with image and interaction prefixes. The main experiments compare bilinear interactions with learned or uniform expert weights, and additive interactions with learned weights.
Critical strengths
-
The analysis goes beyond aggregate benchmark scores and identifies behaviors that matter for interpreting the results. In particular, the late-training gate statistics and the CV-Bench-3D answer distribution expose problems that would be easy to miss from accuracy alone. These observations are useful for understanding what a seemingly reasonable multimodal expert design actually learns.
-
The statistical analysis is careful about what the available evidence can establish. The paired, grouped bootstrap respects the evaluation structure, and the paper distinguishes uncertainty over evaluation examples from variability across training runs. It also avoids treating a confidence interval containing zero as evidence of equivalence. This makes the reported comparisons more credible, even though the experimental scope remains limited.
Critical weaknesses
-
The technical novelty is incremental. Low-rank bilinear fusion, token–region interactions, and attention/convolution-based processing are established ideas. I see the main addition here as using the same spatial interaction map for expert processing, routing, and a direct language-model prefix. This is a reasonable design, but the paper does not establish a clear advantage or capability arising from this particular combination. The three configurations have mixed rankings, and there is no matched simpler connector that would show whether the added structure is worthwhile. As a result, the architecture alone provides a relatively modest contribution, and much of the case for the paper rests on its analysis.
-
The analysis does not yet deliver a sufficiently clear finding beyond the behavior of these particular runs. Nearly uniform gates and small differences between expert-family evaluations are interesting observations, but they leave several explanations open: the experts may provide similar useful features, the decoder may rely mainly on the direct prefixes, or the training procedure may favor uniform weights. The paper discusses these possibilities without distinguishing them. The gate measurements are also taken from training microbatches, while the expert-family comparisons use different code versions. I therefore find it difficult to extract a reliable lesson about when interaction-conditioned routing helps, when it fails, or why. The current study documents a concrete system in detail, but the broader insight remains limited.
Questions for authors
-
What evidence supports complementary roles for the global, local, and hybrid experts? Please distinguish the intended advantage of preserving spatial interaction structure from the more general benefit of conditioning expert weights on both image and text, which is already explored in prior work such as MoVA.
-
For the same B-L checkpoint, what happens when learned gates are replaced by uniform weights, calibration-set mean weights, or gates shuffled across examples? Keeping the expert outputs and other feature paths unchanged would help establish whether input-dependent weighting contributes to the predictions.
-
When the A/B options are swapped and predictions are mapped back to the original meanings, does the model preserve its semantic choices or continue selecting A? This result would materially affect whether I interpret the reported score as evidence about visual reasoning.