acceptodds
Under review as a conference paper at ICLR 2027

MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models

Abstract

Universal multimodal embedding (UME) increasingly demands encoder's capacity for handling a broad range of tasks and modalities with increased complexity. Prior scaling methods either increase the representation size, retrieval effort, or scales the encoder into a heavy multimodal LLM. Recent works, such as Think-Then-Embed (TTE), explore scaling via reasoning tokens. However, embedding models are hard to scale up: increasing parameters directly tradeoffs for the large training batch size that contrastive learning needs, and retrieval has to be served under tight latency. Moreover, UME tasks are diverse in complexity, where scaling up embedders can bring significant redundant computation. In this work, we propose MOEMB, which instead scales UME along the expert axis through mixture-of-experts (MoE), growing encoder capacity while preserving single-vector, non-autoregressive encoding. Through a systematic study of the design space and training recipes for MoE-based UME, MoEMB sets a new state of the art on both MMEB-V2 and MRMR among models trained on public MMEB-family data: with only 3B active parameters, MoEMB surpasses TTE-based methods with >4x active parameters, using significantly less computes. To further improve the scalability and efficiency, we conduct the first comprehensive study of adaptive computation for MoE-based embedding, spanning diverse strategies across training-based and inference-only methods. Together, these results support expert scaling as an effective and efficient direction for UME, with adaptive computation further improving efficiency for MLLM-based embedding models towards large-scale retrieval and recommendation systems.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Embedding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KbNpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MoEMB, which uses a pretrained sparse MoE to scale the encoder capacity of universal multimodal embeddings (UME) while retaining single-vector encoding without generating reasoning text. The authors compare different MoE construction and router adaptation schemes and investigate adaptive computation to further reduce encoding costs. Among the compared models adapted using only public MMEB-family data, the paper achieves leading results on MMEB-V2 and MRMR, demonstrating a favorable trade-off between performance and computational efficiency.

Critical strengths
  1. The problem is important, and the core rationale is clear. The paper addresses a central tension in UME: supporting multiple tasks and modalities requires greater model capacity, whereas large-scale retrieval demands low encoding costs. Expanding capacity through sparse experts without relying on longer reasoning processes is a practically valuable direction. Beyond improving benchmark scores, the work offers a model design direction worth considering for efficient UME.
  2. The study provides useful findings on adaptation and efficiency. The study finds that native MoE outperforms upcycling during downstream adaptation, and that explicit task-based routing does not outperform retaining pretrained routing. For compute reduction, token pruning translates more readily into actual throughput improvements than expert-level sparsity. These findings help explain which designs are effective and which added complexities are unnecessary, giving the work value beyond the results of a single model.
Critical weaknesses
  1. The methodological novelty is relatively limited. Although the authors explore conflict-aware and task-conditioned routing, these additions do not become the main source of the final performance gains. The main model still relies primarily on a pretrained native MoE and contrastive adaptation. The paper's novelty therefore lies more in validating a scaling direction and conducting a systematic study than in introducing a new expert mechanism tailored to UME. Adaptive computation provides additional technical contributions, but these are not sufficient to fully address this limitation.
  2. The central claims are stronger than the conclusions supported by the available evidence. The results strongly support the conclusion that a pretrained native MoE can serve as an efficient UME encoder. However, support remains limited for the general claim that expert scaling is more effective than reasoning scaling. For example, the expert-count curve is obtained by pruning the same pretrained model. Such results demonstrate the importance of learned expert capacity, but make it difficult to disentangle the respective contributions of the MoE architecture, total capacity, and the backbone's pretrained capabilities. This limits the paper's ability to explain the mechanisms behind scaling.
Questions for authors
  1. Based on the existing results, could the authors clarify which evidence supports attributing the performance gains to expert capacity scaling rather than merely adopting a stronger pretrained backbone? Accordingly, under what conditions should the conclusion that "expert scaling outperforms reasoning scaling" be considered valid?
  2. Given that frozen routing already performs well and task-conditioned routing yields no clear gains, is the more plausible explanation that "the pretrained division of labor among experts is already suitable for UME," or that "the current adaptation data are insufficient to effectively reshape this division of labor"? Can the existing analyses help distinguish between these two explanations?
1 comment
Area_Chair_pgx2OpenReview ↗

please examine rebuttal response

Hello Reviewer KbNp,

As you are likely aware, the rebuttal engagement period is coming to a close. Thus, at your earliest convenience, please take a look at the rebuttal and comment and decide if it changes your thoughts regarding your current evaluation.

Thanks.

Reviewer_jZiPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies universal multimodal embedding (UME) using pretrained sparse MoE multimodal backbones. The main approach, MoEMB, adapts Qwen3.5 MoE models into single-vector embedders via standard contrastive InfoNCE training and embedding pooling. The paper also evaluates several router adaptation choices and a broad set of adaptive-computation techniques, including token pruning, adaptive top-k routing, expert skipping/pruning, layer skipping, and early exit. The authors report strong results on MMEB-V2 and MRMR under their restricted training-data setting.

Critical strengths
  1. The adaptive-computation experiments are broad, and the paper makes a useful effort to report actual wall-clock throughput rather than FLOPs alone.
  2. The observation (Figure 10 / Appendix E.2) that cross-task gradient conflict is negligible in UME is a neat empirical finding.
Critical weaknesses

1.The core "method" of this paper is simply taking an existing MoE model (Qwen3.5), slapping a standard contrastive InfoNCE loss on it, and pooling the last token. This is an engineering adaptation rather than a methodological contribution.

2.It is unsurprising that aggressively modifying a router pretrained at large scale for an MoE foundation model may be harmful during relatively limited downstream contrastive fine-tuning. The paper does not provide a stronger MoE-specific mechanism beyond reusing the pretrained experts and router. As such, the gains may be largely attributable to the stronger pretrained Qwen3.5 MoE backbone and its pretraining scale, rather than to MoEMB itself.

Questions for authors

1.What is the total scale of the MMEB-family training data, and is MMEB-family-only training a standard UME protocol?

2.Currently, it is difficult to determine the exact source of the observed improvements. Do the gains arise from (i) the native MoE capacity, (ii) the strength of the Qwen3.5 backbone and its pretraining, (iii) the specific training mixture and recipe, or (iv) avoiding autoregressive reasoning?

3.The adaptive-computation section (token pruning, layer skipping, early exit) appears to be a collection of general VLM acceleration techniques that are not intrinsically related to the MoE architecture. This section reads more like an independent project. Question: Could you clarify if and how these techniques have a specific, intrinsic synergy with your proposed MoE design? If they are largely orthogonal, what is the justification for including them as a core component of this paper's narrative?

1 comment
Area_Chair_pgx2OpenReview ↗

please examine rebuttal response

Hello Reviewer jZiP,

As you are likely aware, the rebuttal engagement period is coming to a close. Thus, at your earliest convenience, please take a look at the rebuttal and comment and decide if it changes your thoughts regarding your current evaluation.

Thanks.

Reviewer_cCUzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoEMB, which scales UME along the expert axis through mixture-of-experts (MoE), growing encoder capacity while preserving single-vector, non-autoregressive encoding. The authors also conduct a systematic exploration of adaptive computation for MoE-based UME embedders and find token pruning to be the most effective adaptive computation approach.

Critical strengths
  • The paper showed that scaling UME along the expert axis through MoE increases encoder capacity with negligible additional compute and provides a better accuracy–throughput trade-off than TTE-based methods.
  • The paper systematically compares different adaptive computation methods and finds that token pruning provides the strongest end-to-end efficiency, while expert-level actions give the weakest.
Critical weaknesses

While MoE is introduced as an efficient way to scale embedding models, the adaptive computation results show that token pruning provides the strongest end-to-end efficiency, whereas expert-level actions give the weakest. The analysis is interesting, but the most effective efficiency improvement comes from token pruning rather than the proposed MoE scaling itself. It may be helpful to better organize the overall flow of the paper so that the motivation and the subsequent analysis form a more coherent story.

Questions for authors

Since token pruning provides the strongest end-to-end efficiency, would similar efficiency gains also be expected for dense UME models?

1 comment
Area_Chair_pgx2OpenReview ↗

please examine rebuttal response

Hello Reviewer cCUz,

As you are likely aware, the rebuttal engagement period is coming to a close. Thus, at your earliest convenience, please take a look at the rebuttal and comment and decide if it changes your thoughts regarding your current evaluation.

Thanks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.