- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposed a tokenizer&embedding construction which allows the LM to be compatible with multiple tokenizer. The main idea is to take the union of vocabulary of each tokenizers as the merged vocab and in the training expose the model to tokenization of each tokenizers. This way the model can work on different tokenization at inference time. The author tested the method on a few small scale pretraining experiement and show that the method only introduce a small degradation.
Critical strengths
None. I don't see crtitical strengths from the work. The method itself is straighteforward and I consider it as a baseline method that most people can come up with when thinking about the problem.
Critical weaknesses
None I don't see critical weakness, but I think there is one main weakness:
the setting of multi-tokenizer joint training is not well-motivated. I would like to understand better why people want to have a model which suppports multiple tokenzier in the same time on the cost of reduced performance. For post-training transfer like FOCUS, the motivation is clear. They might have a new strong model A with tokenzier T_A and an old weak model B with tokenizer T_B. They might want to ensure backward compatbility so it makes sense to have a model A' that works on tokenzier T_B. But train from scratch also limit the model to an existing set of tokenizers, not applicable to future new tokenizers. The author also mentioned in the "Limitations and future work" section that a “universal” draft models for speculative decoding could be a use case. But the trend now in speculative decoding is to move more and more to Multi-Token Prediction or a speculator attached to the model (EAGLE , DFLASH), where the speculator simply can not work universally.
Questions for authors
Q1. One concern is that the problem you are trying to address is not an important problem. Let me know if I'm missing some points. Q2. I have a concern over the evaluation of the "Union-Longest (1×compute)" setting, where you train the model on the joint tokenizer yet your bpb evaluation is on the original routes where the model has never seen during training. This surely will cause a bad performance yet it doesn't bring any information.
6 comments
Following up, response would be appreciated.
Dear Reviewer xssh,
We wanted to briefly follow up on our response and the new experiments in our General Response. In particular, we have now demonstrated a concrete practical application of native multi-tokenizer support through inference-time routing, which was the central motivation concern in your review.
We would appreciate hearing whether these additions address your concerns, and would be happy to discuss any remaining questions.
Reply to (W1 / Q1) On Practical Motivation
Please see my follow-up question posted under your general response on top.
Reply to (Q2) On Union-Longest
Thank you for pointing me to Appendix D. The native evaluation of Union-Longest (Table 12) is exactly the result I was looking for, and it matches what I expected: trained from scratch on its own longest-match segmentation, Union-Longest performs well, reaching 1.0821 BPB. That is lower than String Atlas's five-route mean (1.1027) and than String Atlas's own longest-match route (1.1093).
Given that Union-Longest achieves better BPB, the remaining question is how String Atlas compares to it on the other axis the rebuttal emphasizes, namely compression. Union-Longest can mix strings from all five vocabularies within a single document, so its segmentation may already compress as well as or better than per-document routing. Does String Atlas achieve better compression than Union-Longest? Could the authors add a Union-Longest column to the compression table below, along with Union-Longest's token count on the same mixed-domain stream used in the routing experiment?
| Domain | General | Clean web | Code-skew | Sci/math | Wiki/QA | Union-Longest |
|---|---|---|---|---|---|---|
| CommonCrawl | 0.00% | +0.90% | +5.80% | +4.68% | +7.35% | ? |
| C4 | 0.00% | -3.12% | +4.85% | +4.75% | +7.29% | ? |
| GitHub | 0.00% | +4.69% | -9.56% | +11.42% | -1.82% | ? |
| StackExchange | 0.00% | +0.38% | -7.17% | +6.05% | -5.07% | ? |
| arXiv | 0.00% | +10.00% | +3.81% | -7.35% | +5.39% | ? |
| Wikipedia | 0.00% | -7.95% | +5.89% | -13.66% | -18.62% | ? |
| Configuration | Token savings | Mean BPB |
|---|---|---|
| String Atlas, always General route | — | 1.3793 |
| String Atlas + routing | 8.20% | 1.3964 |
| Union-Longest (native) | ? | ? |
If String Atlas with routing does not compress better than Union-Longest, then a standard model trained on the union vocabulary would be better on both BPB and token count. In that case, the practical case for String Atlas would rest only on preserving the original tokenizer interfaces, and that motivation would need further justification.
Response to Reviewer xssh
Thank you for the review. We address your concerns about the technical contribution, practical motivation, and Union-Longest baseline below. We have also conducted new experiments demonstrating the practical benefits of multi-tokenizer support, summarized in our General Response.
(S1) On the Technical Contribution: We view the simplicity and minimal invasiveness of String Atlas as one of its key strengths.
We do not aim to contribute a complicated new architecture, but rather, a simple change in parameterization that allows independently constructed tokenizers to coexist within a single model. Each tokenizer still keeps its original segmentation rules and next-token distribution, while lexical parameters are shared by exact byte-string identity. The Transformer backbone, training objective, and decoding procedure remain unchanged.
Importantly, this is not equivalent to simply merging vocabularies or sharing a Transformer backbone. Our ablations show that Private Tables, which shares the backbone but not the lexical rows, has a +0.2175 BPB gap to full specialists. Shuffled Atlas, which ties an equal number of unrelated rows, performs even worse (+0.2718). String Atlas gets within just +0.0137 BPB of a full 5x fleet of specialists. Our analysis in Section 7 further establishes that pooling supervision on shared lexical rows, particularly the output rows, is responsible for this improvement.
The same method works across independently trained tokenizers varying in domain, vocabulary size, tokenization algorithm (BPE, WordPiece, and Unigram), and even languages (as we show in the General Response). We consider achieving this generality without modifying the underlying language-model architecture to be an important contribution.
(W1 / Q1) On Practical Motivation:
We agree that the original submission did not sufficiently demonstrate what one gains from supporting multiple tokenizers. We have now directly addressed this with new routing experiments.
As detailed in our General Response, shortest-prefix routing reduces token counts by 8.20% across six text domains and 9.18% across six languages, with BPB increases of only +0.017 and +0.0234, respectively.
This is just one example of a concrete use case. One model can select the tokenizer best suited to each input, rather than committing to a single tokenizer for all domains and languages. No model retraining or weight changes are required when switching among the supported routes.
On Future Tokenizers and Post-Hoc Transfer: We agree that post-hoc methods such as FOCUS are useful when a new tokenizer must be introduced after pretraining. This is a complementary setting; in fact, post-hoc transfer methods can still be applied on top of an existing String Atlas model, due to the minimal nature of our intervention.
In our paper, we mainly addresses the case where multiple tokenizer interfaces are known during pretraining and must all remain available afterward. Our goal is to pretrain one model to natively achieve strong performance on a given set of tokenizers. The practical effects of this decision can be seen throughout our experiments. For example, at 1B scale, zero-shot FOCUS has a mean +0.0823 BPB gap on transferred routes, whereas String Atlas maintains a +0.0126 BPB mean gap because it jointly trains each route (Appendix F).
On Speculative Decoding: Universal draft models were one possible future application we highlighted in the original paper. More importantly, we have now demonstrated a concrete practical application through inference-time tokenizer routing. As detailed in our General Response, a simple routing policy reduces token counts by 8.20% across text domains and 9.18% across languages, with only minimal increases in BPB.
(Q2) On Union-Longest:
We agree that evaluating Union-Longest only under the original tokenizer routes does not characterize its native performance. This is precisely why Appendix D reports both evaluations.
The fully trained Union-Longest model achieves 1.0821 BPB under its native longest-match tokenizer, but 1.2688 BPB under the five original tokenizer routes. The point is not that Union-Longest is a poor tokenizer by itself. Our goal was to show that training with a merged vocabulary under one segmentation does not provide calibrated support for the original tokenizers simply by restricting the vocabulary (even when every token string is available).
Thank you again for the thorough review; we would be glad to discuss any of these points further.