acceptodds
Under review as a conference paper at ICLR 2027

SG-MoE: Shared Multi-Granularity MoE with MLLM-Assisted Semantic Router Learning for Cross-Modal Representation Learning and Zero-Shot Retrieval

Abstract

Zero-shot cross-modal retrieval matches semantically related samples across modalities under disjoint training and test categories, yet existing methods face insufficient multi-granularity coordination, limited cross-modal alignment beyond output representations, and semantically ambiguous expert routing. To address these issues, we propose shared multi-granularity MoE with MLLM-assisted semantic router learning for cross-modal representation learning and zero-shot retrieval (SG-MoE). We design a dual-stream MoE to learn fine-grained matching and semantic-level representations; introduce modality-shared experts to constrain internal cross-modal transformations while retaining modality-specific routers; and develop MLLM-assisted semantic router learning with an offline MLLM-built ontology and frozen-encoder supervision during training. Across four benchmarks, SG-MoE improves over task-specific baselines; ablations identify dual-stream modeling and modality-shared experts as the main retrieval contributors, while routing analyses support concept–expert organization rather than substantial Avg mAP gains. Retrieval uses the trained SG-MoE routers with fixed zero semantic input and no external MLLM or vision–language encoder.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Zero-Shot Cross-Modal Retrieval, Cross-Modal Representation Learning, Mixture-of-Experts, Multi-Granularity Learning, Cross-Modal Alignment, Semantic Router Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qRDVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SG-MoE for zero-shot cross-modal retrieval, combining dual-stream multi-granularity learning, modality-shared experts, and MLLM-assisted semantic routing. Experiments on four benchmarks show improvements over task-specific baselines.

Critical strengths
  1. Reasonable design. The dual-stream architecture captures both fine-grained and semantic information, while shared experts help align different modalities.
  2. Solid experiments. The method is tested on four benchmarks, and the ablation studies provide useful evidence for the main components.
Critical weaknesses
  1. Limited impact of semantic routing. Removing semantic router learning barely changes Avg mAP (0.6027 vs. 0.6024), making its actual contribution unclear.
  2. Modest gains from MoE. The improvement over a single expert is relatively small. Without parameter-matched comparisons, it is hard to tell whether the gains come from expert routing or simply more model capacity.
  3. Training-inference mismatch. The routers use semantic concept distributions during training but receive zero vectors at inference. The paper does not clearly explain how this affects performance.
Questions for authors
  1. Given the small performance difference, what benefits does MLLM-assisted semantic routing offer beyond interpretability?
  2. Could the authors provide a parameter-matched comparison with the single-expert baseline to better justify the MoE design?
Reviewer_zdSYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose SG-MoE, a method to retrieve images from text and text from images when the test categories differ from those used for training. The method combines selected local details with information from the complete input and learns to choose between processing modules shared by images and text using a mixture of experts.

Critical strengths

The overall idea of investigating sharing processing modules between images and text is sensible.

Critical weaknesses
  • W1. The main weakness is that it remains unclear to me why the proposed method is needed. The introduction lists several limitations of existing approaches. However, the experiments do not show that these limitations lead to worse retrieval performance or that the proposed components address them. SG-MoE performs worse than CLIP on three of four datasets and improves on only one. I agree that a method does not have to outperform CLIP everywhere to be useful, but then I would expect a clearer benefit elsewhere, such as lower computational cost or better explanations of retrieval decisions. The paper does not establish either sufficiently to justify the additional complexity.
  • W2. The ablations also highlight my concern about the nessecity of the sopisticatedness of the method. Removing semantic router learning leaves average retrieval performance essentially unchanged, and the single-expert version already achieves most of the full model's performance. Combining the two representations and sharing parameters appear useful, but the benefit of the routing mechanism is less clear. Can the authors compare against a simpler model with these two representations, comparable capacity, and the same training supervision? Because the current comparison does not isolate the improvement from selecting different experts for different inputs.
  • W3. Several choices of the method are neither sufficiently motivated nor evaluated. Training uses cosine similarity for both representations and learns how to combine them for each query. At inference, one representation uses Euclidean distance instead, and a fixed weight replaces the learned combination. Why is the model trained with a scoring rule that is then discarded? Also the method adds several auxiliary losses and a separate teacher for the semantic representation, whose contributions remain unclear to me. I do not think every implementation detail needs an ablation, but at least the motivation for each choice should be given.
  • W4. When the paper gives a motivation or interpretation, it is too strong. For example, selecting high-scoring elements does not by itself establish fine-grained discrimination, and aggregating the complete input does not make a representation semantic or more transferable. Or similarly, sharing expert parameters constrains the transformations but does not demonstrate internal semantic alignment.
  • W5. The routing analysis shows stronger associations between concepts and expert assignments, but it does not show that the experts perform distinct computations or explain the retrieval decisions. For example, assigning an animal image to an expert called "Animal" does not establish that this expert contributes animal-specific information to retrieval. Can the authors test whether removing or exchanging an expert selectively affects retrieval for its associated concepts?
  • W6. Several parts of the method are insufficiently specified. These include the construction and dimensions of the input sequence representations, differentiable top-k pooling, the "representative" paths used during curriculum warm-up, the score blocks and normalization axes used at inference, the auxiliary-loss weights, the routing purity and entropy, and how the semantic representation is made compatible with the teacher's class distribution.
  • W7. Finally the writing makes the paper difficult to follow. Long component names are repeated where a direct explanation of the operation would be more useful. Terms such as "multi-granularity coordination", "semantic-level representations", and "alignment beyond the output space" appear before their meaning is clear. I often had to work backward from the equations to understand what the authors meant. The paper would be easier to understand if it explained the computation directly, introduced notation when it is needed, and clearly separated training from inference.
Questions for authors

Please see weaknesses

Reviewer_sZcwOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SG-MoE, a shared multi-granularity mixture-of-experts framework for zero-shot cross-modal retrieval. It introduces two parallel MoE streams to capture fine-grained discriminative features and global semantic representations, while sharing expert parameters across image and text modalities to encourage internal cross-modal alignment. Additionally, an MLLM-assisted semantic router learning strategy is proposed, using an offline concept ontology and frozen vision-language encoder supervision to organize expert routing. Experiments on four cross-modal retrieval benchmarks demonstrate improvements over several task-specific baselines.

Critical strengths

S1. The paper presents a reasonable approach to multi-granularity representation learning for zero-shot cross-modal retrieval. The dual-stream MoE jointly captures local discriminative information and global semantic representations.

S2. Sharing expert parameters across image and text modalities while retaining modality-specific routers is an interesting design that constrains cross-modal transformations without removing modality-specific flexibility.

S3. The experiments cover four benchmarks and include component ablations and routing interpretability analyses, providing useful evidence for the proposed architecture.

Critical weaknesses

W1. The practical contribution of semantic router learning is unclear. Table 4 shows that removing Semantic Router Learning changes Avg mAP only from 0.6027 to 0.6024, indicating almost no retrieval performance difference. This suggests that its main contribution may be improving concept-expert organization rather than actual retrieval performance, raising questions about its necessity and significance as a central component of the proposed method.

W2. There is a discrepancy between training and inference. During training, routers are conditioned on semantic concept distributions, and the retrieval branches are optimized using sample-dependent fusion weights. However, inference replaces semantic inputs with fixed zero vectors and uses a validation-selected fixed fusion weight with different similarity formulations. The impact of this train-test discrepancy on generalization is not sufficiently investigated.

W3. The claimed internal cross-modal semantic alignment is insufficiently validated. Sharing expert parameters constrains image and text representations to use the same transformations, but this does not necessarily guarantee semantic consistency between their intermediate representations. The current experiments mainly demonstrate improvements in retrieval performance and do not provide direct evidence of improved internal cross-modal alignment.

Questions for authors
  1. Given that Semantic Router Learning contributes almost no improvement in Avg mAP, what is its main practical benefit? Can the authors demonstrate that semantic routing provides meaningful gains in zero-shot generalization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.