acceptodds
Under review as a conference paper at ICLR 2027

MoVE: Mixture-of-Vocabulary-Experts for Improved Representation Learning

Abstract

Vocabulary size is a key design choice in transformers, with recent work showing that larger models benefit from larger vocabularies and achieve better performance at the same training cost. Expanding the output vocabulary is costly due to the softmax computation while input vocabulary scaling is relatively inexpensive as it only involves lookup. However, scaling input vocabulary presents a big challenge—a long tail of rare tokens in the training data receive fewer gradient updates, remain under-trained, and can also adversely impact embedding quality. This is particularly problematic for encoder-only text embedding models, which depend on high-quality token embeddings to build document representations. We present Mixture-of-Vocabulary-Experts (MoVE) architecture for training encoder-only models that enables effective vocabulary scaling by combining tokens of a small base vocabulary to generate a much larger vocabulary, with the help of a Mixture-of-Experts layer. Each expert specializes in a subset of the expanded vocabulary and can generate the trained embeddings offline, leading to no additional overhead during inference. Strong empirical results across the MTEB benchmark show that MoVE trained with vocabulary sizes up to consistently outperforms naive vocabulary scaling, comparable baselines as well as deeper models with base vocabulary, while maintaining lower inference latency.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vocabulary scaling, mixture-of-experts, dense retrieval, encoder-only models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4idZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new method to scale the vocabulary of LM. The naive way of scaling vocabulary is to have a larger BPE vocab yet it implies larger embedding layer and also potentially low training signal for rare tokens. The author suggests introducing a compositional module to compute the embedding of extra merge tokens at runtime (training only, inference-time only does look up ). The method here uses a MoE structure, routing different tokens to different experts to compute the embedding. The compositional approch should alleviate the problem of rare tokens. And the method also reduces the number of tokens actually fed into the transformer layers by using joint embeddings, which can improve inference efficiency. The author evaluated the method against a few baselines with BERT models on MTEB benchmarks. The two evlauation metrics are accuray are latency.

Critical strengths

None. I don't see a critical strength, but the most interesting technique presented in the work is to use the merged token embedding for the router and the constituent token embeddings for the expert to combine.

Critical weaknesses

Two critical weakness:

  1. How much this work differs from existing works like SCONE (https://arxiv.org/abs/2502.01637)? SCONE seems the most similar and direct baseline for this work. While the author listed 4 points of differences in the related work section, the only one which is really convincing is the sequence length argument: SCONE doesn't actually reduce seq length yet this work does. However, it seems that SCONE can be trivially extended to reduce seq length as well, in particularly in the encoder-only architecture where there is no concern over the output embedding layer. While I acknowlege the work does make an interesting design of bringing the MoE, it doesn't really solve something existing works can not solve.
  2. The results presented in table-1 show that MoVE beats SCONE and other methods, yet at a very small margin. The margin is so small that I can not rule out the possbility of pure variation across runs. The author should provide std across runs.
Questions for authors
  1. I would like the author to clarify how they differ from SCONE in a more rigorous way.

Both Huang et al. (2025) and Yu et al. (2025) rely on n-gram/f-gram expansions, which mainly create merged tokens rather than new lexical units. Our approach instead builds larger vocabularies directly from corpus statistics (e.g., extended BPE merges), yielding genuinely new tokens and broader coverage of rare words.

I don't see how is "new lexical units" different from "merged tokens", they are the same thing afterall. If your method has a better broader coverage of rare words, does it improve the performance over these words? This not convincing.

We would also like to indicate that introducing a small transformer to model the f-grams dramatically increases the parameter count which requires more number of training points to be effectively trained.

Yes, but this advantage is marginal given the models tested in the work are all small models

Additionally, since the sequence length is retained, there are limited latency speedups. On the other hand, MoVE can provide lower inference latency and function well in low data regimes (see Appendix A.3 and Appendix C.1).

This makes sense, but as I explained above, SCONE can be easily adapted to reduce seq lenegth as well.

Moreover, in SCONE, a small transformer model is meant to handle all the expanded vocabulary token combinations ranging around 500k, while the MoE design distributes this load, leading to better accuracy.

You need to support the "better accuracy" argument. The improvements shown in Table-1 is too small.

  1. The method doesn't show a big enough improvement over the existing methods. This per se is not a reason to reject a paper, but as this work doesn't bring theoretical insights or insipiration for future works, I think its contribution is not enough.
Reviewer_1seNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a significant issue in the expansion of encoder input vocabularies: while a large vocabulary can shorten input sequences, the under-training of low-frequency tokens may compromise representation quality. The authors propose MoVE, which employs a mixture-of-experts approach to combine vectors from a smaller vocabulary, thereby enabling a large vocabulary for encoder-only text embedding models and mitigating the problem of under-trained rare tokens. The research motivation is clear, effectively balancing representation quality with inference efficiency, which gives the work genuine research value. Furthermore, the design that utilizes offline generation combined with online lookup offers excellent practical utility for deployment.

Critical strengths
  1. This research is highly significant. Given that the primary limitations of large language models are efficiency and cost, this work directly contributes to improving the efficiency of such models.
  2. The experiments are extensive.
Critical weaknesses
  1. There is insufficient direct evidence to support the claim that the under-training of low-frequency tokens is mitigated. While Figures 1 and 5 illustrate the relationships between vocabulary size, data volume, performance, and average token frequency, they do not explicitly confirm that performance degradation is primarily driven by the under-training of "tail" (low-frequency) tokens. Similarly, although Figure 7 and Table 13 report overall performance gains, they do not analyze the model's behavior specifically on samples involving low-frequency tokens. Furthermore, the significant changes in embedding norms observed in Figure 8 cannot be taken as definitive proof of improved semantic representation, as the updating of shared parameters inherently drives shifts in the vectors of low-frequency tokens. It is recommended to conduct an analysis using a consistent vocabulary size and tokenization method: categorize evaluation queries into low-, medium-, and high-frequency groups based on the training frequency of the tokens they contain. After controlling for query length, compare the nDCG@10 scores—and the relative performance differences—between MoVE and a baseline model with a large, static vocabulary. This would help verify whether the performance gains are indeed more pronounced for the low-frequency group.
  2. There are unexplained inconsistencies in the experimental results under low-data conditions, which undermines the verifiability of the claims regarding sample efficiency. Although Appendix C.1 states that the settings from Appendix A.3 are used, the reported results differ between the two sections: for the 1M-data, 5k→100k configuration, MoVE achieves a score of 34.87 in Figure 6(b) but 35.95 in Figure 7; for the 2M-data setting, the scores are 37.63 and 38.67, respectively. Similar discrepancies appear in the results for the 200k-vocabulary expansion and the 5k-vocabulary baseline, yet the text provides no explanation for these variations in experimental setup.
Questions for authors
  1. This method was only evaluated on encoder-only architectures; it has not been tested on decoder-only architectures. I am curious whether this method would also be effective for decoder-only architectures.
  2. In Table 3, latency is measured exclusively on the MS MARCO dataset. Would it be better to include additional datasets? This is because the nature of the tokens themselves could also influence efficiency.
Reviewer_JCUmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Mixture-of-Vocabulary-Experts (MoVE), an architecture for scaling input vocabularies in encoder-only models while addressing the under-training of rare tokens. The method uses a small base vocabulary and a mixture-of-experts layer to construct embeddings for a larger vocabulary, with the expert computations performed offline for inference. Experiments on 47 MTEB tasks using 4-layer and 12-layer BERT models show consistent improvements over naive vocabulary scaling and existing approaches such as SCONE and Over-Encoding. MoVE also achieves lower inference latency through shorter token sequences.

Critical strengths

The paper addresses a practical and relevant problem: the challenge of scaling input vocabularies without sacrificing embedding quality. The proposed expert-based vocabulary composition is an interesting approach to improving the training of rare token embeddings. The paper also conducted comprehensive empirical evaluation. The method consistently improves naive vocabulary scaling across different model sizes and demonstrates competitive performance against existing vocabulary expansion techniques. The ablations and latency measurements further support its effectiveness.

Critical weaknesses

The paper shows relatively modest improvements over strong baselines. Although MoVE substantially outperforms naive vocabulary scaling, its improvement over SCONE is relatively small (0.27 points on the overall MTEB score for BERT-4L at 500k vocabulary). This raises questions about whether the additional architectural complexity is justified. The paper also shows incomplete efficiency comparison. The reported inference speedup primarily comes from shorter token sequences, rather than the MoVE architecture itself. Furthermore, the expanded vocabulary substantially increases model storage and CPU memory requirements. A more complete comparison of accuracy, latency, memory, and training cost is needed to establish the practical benefits.

Questions for authors

1/ Could the authors provide a more detailed comparison of MoVE and existing methods under matched parameter budgets and training compute? This would help establish how much improvement comes specifically from the proposed architecture. 2/ Since the inference speedup largely comes from shorter sequences, how does MoVE compare against naive large-vocabulary models in inference latency and memory consumption? 3/ Could the authors provide a more direct evaluation of rare-token representation quality? The current analysis of embedding norm changes is suggestive, but does not necessarily demonstrate improved semantic representations for rare tokens.

Comments on the paper

1 comment

Dear Reviewer,

Thank you for your efforts in evaluating this submission. The authors have uploaded their reply. Please take your responsibility to review the authors' response, revisit your evaluation,and update your score and recommendation as appropriate. Please add comments on whether the authors have sufficiently addressed your raised points. If you have further questions,please discuss them with the authors.

Thanks,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.