- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a new method to scale the vocabulary of LM. The naive way of scaling vocabulary is to have a larger BPE vocab yet it implies larger embedding layer and also potentially low training signal for rare tokens. The author suggests introducing a compositional module to compute the embedding of extra merge tokens at runtime (training only, inference-time only does look up ). The method here uses a MoE structure, routing different tokens to different experts to compute the embedding. The compositional approch should alleviate the problem of rare tokens. And the method also reduces the number of tokens actually fed into the transformer layers by using joint embeddings, which can improve inference efficiency. The author evaluated the method against a few baselines with BERT models on MTEB benchmarks. The two evlauation metrics are accuray are latency.
Critical strengths
None. I don't see a critical strength, but the most interesting technique presented in the work is to use the merged token embedding for the router and the constituent token embeddings for the expert to combine.
Critical weaknesses
Two critical weakness:
- How much this work differs from existing works like SCONE (https://arxiv.org/abs/2502.01637)? SCONE seems the most similar and direct baseline for this work. While the author listed 4 points of differences in the related work section, the only one which is really convincing is the sequence length argument: SCONE doesn't actually reduce seq length yet this work does. However, it seems that SCONE can be trivially extended to reduce seq length as well, in particularly in the encoder-only architecture where there is no concern over the output embedding layer. While I acknowlege the work does make an interesting design of bringing the MoE, it doesn't really solve something existing works can not solve.
- The results presented in table-1 show that MoVE beats SCONE and other methods, yet at a very small margin. The margin is so small that I can not rule out the possbility of pure variation across runs. The author should provide std across runs.
Questions for authors
- I would like the author to clarify how they differ from SCONE in a more rigorous way.
Both Huang et al. (2025) and Yu et al. (2025) rely on n-gram/f-gram expansions, which mainly create merged tokens rather than new lexical units. Our approach instead builds larger vocabularies directly from corpus statistics (e.g., extended BPE merges), yielding genuinely new tokens and broader coverage of rare words.
I don't see how is "new lexical units" different from "merged tokens", they are the same thing afterall. If your method has a better broader coverage of rare words, does it improve the performance over these words? This not convincing.
We would also like to indicate that introducing a small transformer to model the f-grams dramatically increases the parameter count which requires more number of training points to be effectively trained.
Yes, but this advantage is marginal given the models tested in the work are all small models
Additionally, since the sequence length is retained, there are limited latency speedups. On the other hand, MoVE can provide lower inference latency and function well in low data regimes (see Appendix A.3 and Appendix C.1).
This makes sense, but as I explained above, SCONE can be easily adapted to reduce seq lenegth as well.
Moreover, in SCONE, a small transformer model is meant to handle all the expanded vocabulary token combinations ranging around 500k, while the MoE design distributes this load, leading to better accuracy.
You need to support the "better accuracy" argument. The improvements shown in Table-1 is too small.
- The method doesn't show a big enough improvement over the existing methods. This per se is not a reason to reject a paper, but as this work doesn't bring theoretical insights or insipiration for future works, I think its contribution is not enough.
Dear Reviewer,
Thank you for your efforts in evaluating this submission. The authors have uploaded their reply. Please take your responsibility to review the authors' response, revisit your evaluation,and update your score and recommendation as appropriate. Please add comments on whether the authors have sufficiently addressed your raised points. If you have further questions,please discuss them with the authors.
Thanks,
Area Chair