acceptodds
Under review as a conference paper at ICLR 2027

String Atlas: Breaking the One-Tokenizer Assumption in Language Modeling

Abstract

A standard language model is trained with a single canonical tokenizer, so that tokenizer is effectively frozen into its weights. This structural constraint means that changing the tokenizer afterwards invalidates the learned embedding parameters connecting raw text to the Transformer trunk. Yet, tokenizer choice is far from a cosmetic preprocessing step; it dictates domain compression, effective context length, arithmetic capabilities, and multilingual fairness. While existing methods either commit to one universal vocabulary upfront or resort to expensive post-hoc vocabulary transfer, we introduce String Atlas, an architecture that natively supports multiple tokenizers from pretraining. Instead of equipping each tokenizer with private embedding tables, which fragments learning, String Atlas keys its lexical parameters directly to the emitted byte string. Tokenizers may disagree on boundaries, but they seamlessly share rows in a common atlas whenever they emit the same string. Across suites of tokenizer sets varying domains, granularity, and even algorithm, we find that String Atlas allows a language model to support several tokenizer interfaces simultaneously, nearly matching the performance of independently trained specialists while maintaining the training and inference costs of a single model. Together, these results break the one-tokenizer assumption that binds a model to its vocabulary and reframe tokenizers as interchangeable views over a single shared lexicon.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Tokenization, Pre-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xsshOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed a tokenizer&embedding construction which allows the LM to be compatible with multiple tokenizer. The main idea is to take the union of vocabulary of each tokenizers as the merged vocab and in the training expose the model to tokenization of each tokenizers. This way the model can work on different tokenization at inference time. The author tested the method on a few small scale pretraining experiement and show that the method only introduce a small degradation.

Critical strengths

None. I don't see crtitical strengths from the work. The method itself is straighteforward and I consider it as a baseline method that most people can come up with when thinking about the problem.

Critical weaknesses

None I don't see critical weakness, but I think there is one main weakness:

the setting of multi-tokenizer joint training is not well-motivated. I would like to understand better why people want to have a model which suppports multiple tokenzier in the same time on the cost of reduced performance. For post-training transfer like FOCUS, the motivation is clear. They might have a new strong model A with tokenzier T_A and an old weak model B with tokenizer T_B. They might want to ensure backward compatbility so it makes sense to have a model A' that works on tokenzier T_B. But train from scratch also limit the model to an existing set of tokenizers, not applicable to future new tokenizers. The author also mentioned in the "Limitations and future work" section that a “universal” draft models for speculative decoding could be a use case. But the trend now in speculative decoding is to move more and more to Multi-Token Prediction or a speculator attached to the model (EAGLE , DFLASH), where the speculator simply can not work universally.

Questions for authors

Q1. One concern is that the problem you are trying to address is not an important problem. Let me know if I'm missing some points. Q2. I have a concern over the evaluation of the "Union-Longest (1×compute)" setting, where you train the model on the joint tokenizer yet your bpb evaluation is on the original routes where the model has never seen during training. This surely will cause a bad performance yet it doesn't bring any information.

6 comments
AuthorsOpenReview ↗

Response to Reviewer xssh

Thank you for the review. We address your concerns about the technical contribution, practical motivation, and Union-Longest baseline below. We have also conducted new experiments demonstrating the practical benefits of multi-tokenizer support, summarized in our General Response.

(S1) On the Technical Contribution: We view the simplicity and minimal invasiveness of String Atlas as one of its key strengths.

We do not aim to contribute a complicated new architecture, but rather, a simple change in parameterization that allows independently constructed tokenizers to coexist within a single model. Each tokenizer still keeps its original segmentation rules and next-token distribution, while lexical parameters are shared by exact byte-string identity. The Transformer backbone, training objective, and decoding procedure remain unchanged.

Importantly, this is not equivalent to simply merging vocabularies or sharing a Transformer backbone. Our ablations show that Private Tables, which shares the backbone but not the lexical rows, has a +0.2175 BPB gap to full specialists. Shuffled Atlas, which ties an equal number of unrelated rows, performs even worse (+0.2718). String Atlas gets within just +0.0137 BPB of a full 5x fleet of specialists. Our analysis in Section 7 further establishes that pooling supervision on shared lexical rows, particularly the output rows, is responsible for this improvement.

The same method works across independently trained tokenizers varying in domain, vocabulary size, tokenization algorithm (BPE, WordPiece, and Unigram), and even languages (as we show in the General Response). We consider achieving this generality without modifying the underlying language-model architecture to be an important contribution.


(W1 / Q1) On Practical Motivation:

We agree that the original submission did not sufficiently demonstrate what one gains from supporting multiple tokenizers. We have now directly addressed this with new routing experiments.

As detailed in our General Response, shortest-prefix routing reduces token counts by 8.20% across six text domains and 9.18% across six languages, with BPB increases of only +0.017 and +0.0234, respectively.

This is just one example of a concrete use case. One model can select the tokenizer best suited to each input, rather than committing to a single tokenizer for all domains and languages. No model retraining or weight changes are required when switching among the supported routes.

On Future Tokenizers and Post-Hoc Transfer: We agree that post-hoc methods such as FOCUS are useful when a new tokenizer must be introduced after pretraining. This is a complementary setting; in fact, post-hoc transfer methods can still be applied on top of an existing String Atlas model, due to the minimal nature of our intervention.

In our paper, we mainly addresses the case where multiple tokenizer interfaces are known during pretraining and must all remain available afterward. Our goal is to pretrain one model to natively achieve strong performance on a given set of tokenizers. The practical effects of this decision can be seen throughout our experiments. For example, at 1B scale, zero-shot FOCUS has a mean +0.0823 BPB gap on transferred routes, whereas String Atlas maintains a +0.0126 BPB mean gap because it jointly trains each route (Appendix F).

On Speculative Decoding: Universal draft models were one possible future application we highlighted in the original paper. More importantly, we have now demonstrated a concrete practical application through inference-time tokenizer routing. As detailed in our General Response, a simple routing policy reduces token counts by 8.20% across text domains and 9.18% across languages, with only minimal increases in BPB.


(Q2) On Union-Longest:

We agree that evaluating Union-Longest only under the original tokenizer routes does not characterize its native performance. This is precisely why Appendix D reports both evaluations.

The fully trained Union-Longest model achieves 1.0821 BPB under its native longest-match tokenizer, but 1.2688 BPB under the five original tokenizer routes. The point is not that Union-Longest is a poor tokenizer by itself. Our goal was to show that training with a merged vocabulary under one segmentation does not provide calibrated support for the original tokenizers simply by restricting the vocabulary (even when every token string is available).


Thank you again for the thorough review; we would be glad to discuss any of these points further.

AuthorsOpenReview ↗

Following up, response would be appreciated.

Dear Reviewer xssh,

We wanted to briefly follow up on our response and the new experiments in our General Response. In particular, we have now demonstrated a concrete practical application of native multi-tokenizer support through inference-time routing, which was the central motivation concern in your review.

We would appreciate hearing whether these additions address your concerns, and would be happy to discuss any remaining questions.

Reply to (W1 / Q1) On Practical Motivation

Please see my follow-up question posted under your general response on top.

Reply to (Q2) On Union-Longest

Thank you for pointing me to Appendix D. The native evaluation of Union-Longest (Table 12) is exactly the result I was looking for, and it matches what I expected: trained from scratch on its own longest-match segmentation, Union-Longest performs well, reaching 1.0821 BPB. That is lower than String Atlas's five-route mean (1.1027) and than String Atlas's own longest-match route (1.1093).

Given that Union-Longest achieves better BPB, the remaining question is how String Atlas compares to it on the other axis the rebuttal emphasizes, namely compression. Union-Longest can mix strings from all five vocabularies within a single document, so its segmentation may already compress as well as or better than per-document routing. Does String Atlas achieve better compression than Union-Longest? Could the authors add a Union-Longest column to the compression table below, along with Union-Longest's token count on the same mixed-domain stream used in the routing experiment?

DomainGeneralClean webCode-skewSci/mathWiki/QAUnion-Longest
CommonCrawl0.00%+0.90%+5.80%+4.68%+7.35%?
C40.00%-3.12%+4.85%+4.75%+7.29%?
GitHub0.00%+4.69%-9.56%+11.42%-1.82%?
StackExchange0.00%+0.38%-7.17%+6.05%-5.07%?
arXiv0.00%+10.00%+3.81%-7.35%+5.39%?
Wikipedia0.00%-7.95%+5.89%-13.66%-18.62%?
ConfigurationToken savingsMean BPB
String Atlas, always General route—1.3793
String Atlas + routing8.20%1.3964
Union-Longest (native)??

If String Atlas with routing does not compress better than Union-Longest, then a standard model trained on the union vocabulary would be better on both BPB and token count. In that case, the practical case for String Atlas would rest only on preserving the original tokenizer interfaces, and that motivation would need further justification.

Reviewer_Ja4JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

A language model is locked to the tokenizer it was pretrained with, but different tokenizers can suit different domains. The paper outlines a method of including multiple tokenizers in LLM pretraining using byte strings to key embedding and unembedding rows instead of a tokenizer ID. This means that several tokenizers can share one table, with any string they have in common forming one row. The authors claim that such a model performs almost as well as independently trained per-tokenizer specialists, but with the advantage of being a single model. They investigate this across domains, vocabulary sizes and algorithm variants, concluding that the improvement is due to pooling supervision on shared rows rather than the shared transformer trunk.

Critical strengths
  1. The method is minimally invasive and not overly complicated. It is an elegant, one-line concept that uses byte strings to key the embedding/unembedding rows instead of arbitrary token IDs. This means that every tokenizer becomes a view of one table, with no new parameters except the unified embedding matrix and no additional overhead in the forward pass.

  2. The experiments and their setup are well designed. The ablations cleanly isolate the mechanism by investigating private tables, shuffled tables, and input- and output-only tables. They also conduct a row-copy test and plot exposure curves.

  3. The Atlas method appears consistent across domains, granularities, and algorithm suites, and is effective at 86M, 500M, and 1B scales. It also matches the specialists on downstream applications and beats the one-model baselines.

Critical weaknesses
  1. One central motivating premise (that domain tokenizers compress their domains better, see Table 1) is never tested: all language model training and evaluation is based on FineWeb-Edu, a corpus on which the compression of the five tokenizers was not measured. For example, the code-skew route in the Atlas model is never evaluated on code.

  2. The method itself does not increase the compute required for the forward pass, but real-world auto-regressive inference with an Atlas model requires routes to be chosen in advance. Demonstrating the advantages of the Atlas method by selecting the correct route for the correct task would nicely address the paper's motivation. The route selection policy is the crucial ingredient for this, but sadly it is left to future work. Evaluating each route on its matching domain, or even some simple route-selection experiments, would, in my opinion, close the loop between your motivation and conclusion nicely, and I would adjust my score accordingly.

  3. Although I count the method's minimality as an advantage, it is still from-scratch only and requires full-sized pretraining every time a new tokenizer is added. The paper does not present or conduct experiments involving the addition of a route to an existing model. Investigating and/or discussing this would clarify the feasibility of this method for the reader and strengthen the paper.

  4. There is no related work section in the main text. App A contains the entire related work section, but is never referenced. Section 2 motivates the tokenizer suite (including one large table and figure), but it does not adequately position the Atlas method itself against past and current trends in research. Also, terminology is inconsistent and confusing: the shared rows, the tokenizers, and even table columns go by different names (e.g. lexical rows / atlas rows / token parameters, route / view / interface, gap / BPB tax / quality tax), which made the paper harder to read than it needs to be.

Questions for authors
  1. What is the LM quality on the actual domains (code, arxiv, wiki) per route? Does code-skew beat general on github text?

  2. Can a route be added to a trained Atlas model through continued pre-training? Is full parameter training required, and how much data is needed? What is your intuition here? Do you already have some interesting initial evidence?

  3. What are the per-task and per-route downstream numbers (Table 15/16)?

  4. Why is 'Private Tables' (+0.2175) worse than 'Compute-Matched Specialists' (+0.1991), even though its trunk sees 5x the tokens? The paper only says that transformer trunk is not the source of the gain.

  5. Are all the results single runs? If so, what is the seed variance of the BPB gap at 86M?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Ja4J [1/2]

Thank you for the thoughtful review and constructive suggestions. In particular, you suggested evaluating tokenizer routes on their matching domains and demonstrating a practical route-selection policy. We have now completed both experiments, with results summarized in our General Response. We address your individual points below.

(W1 / Q1) On Domain-Specific LM Quality: We now evaluate all five routes of our 1B-scale String Atlas model on held-out text from the six text domains. The table below reports BPB (lower is better):

DomainGeneralClean webCode-skewSci/mathWiki/QA
CommonCrawl1.01251.00951.01881.01511.0161
C41.04271.04161.03561.03711.0431
GitHub1.27571.34931.30291.30511.2975
StackExchange1.40691.41921.41621.41901.4063
arXiv1.80231.77931.79251.82181.7757
Wikipedia1.68251.70471.69271.72281.7482

Somewhat counterintuitively, the tokenizer that compresses a domain best does not always achieve the best BPB. For example, Code-skew reduces GitHub token counts by 9.56%, but General achieves better BPB (1.2757 vs. 1.3029). We observe the same pattern with independently trained full specialists. However, this is expected because our language models were pretrained on FineWeb-Edu, rather than the individual domains.

(W2) On Practical Route Selection: Following your suggestion, we tested a simple shortest-prefix routing policy. For each document, we select the tokenizer that compresses a short prefix best, then use that route for the full document. This reduces token counts by 8.20%, with only +0.017 BPB in additional loss.

We also extended this experiment to multilingual pretraining, where the same policy achieves 9.18% fewer tokens at +0.0234 BPB across six languages. See our General Response for the full results.

These experiments directly demonstrate the practical benefit of switching tokenizer routes at inference time, which we believe addresses the central motivation you identified as missing from the original submission.

(W3 / Q2) On Adding Tokenizers After Pretraining: While our experiments focus on tokenizers specified before pretraining, String Atlas is fully compatible with existing post-hoc tokenizer transfer methods. Adding a new route does not require changing the architecture or necessarily retraining from scratch.

One can simply initialize the new tokenizer's lexical parameters using a method such as FOCUS, then continue pretraining with the additional route. Existing byte strings would reuse their lexical rows from the atlas, while new strings can be initialized through transfer. The existing tokenizer routes can also be included during continued training to preserve their performance.

We hypothesize, however, that with String Atlas, one could also devise a more clever transfer approach making use of the multiple existing tokenizer routes. One could transfer from the tokenizer route with most overlap, combine information from multiple routes, or use the entire shared atlas as the source vocabulary. We leave this as an interesting direction for future work. In our setting, the important part is that existing post-hoc transfer machinery can be applied directly to a String Atlas-trained model once a base route is selected.

(Q3) On Per-Task and Per-Route Downstream Results: This is a valuable addition. We have revised the paper to include this (new sections can be seen marked in blue). Appendix F.2 now reports both per-task and per-route downstream results for our main five-tokenizer suite.

(Q4) On Private Tables vs. Compute-Matched Specialists: This is an interesting phenomenon at small scale, but the pattern reverses at larger scales:

ScaleSpecialists (CM) gapPrivate Tables gapString Atlas gap
86M+0.1991+0.2175+0.0137
500M+0.1959+0.1272+0.0129
1B+0.1427+0.1135+0.0126

At 86M, it seems that sharing the backbone does not compensate for the difficulty of jointly learning five tokenizer routes with separate lexical tables. At larger scales, backbone sharing becomes beneficial, and Private Tables outperforms compute-matched specialists.

However, Private Tables still fragments lexical supervision across tokenizer-local rows, which explains why its performance remains substantially worse than String Atlas.

AuthorsOpenReview ↗

Response to Reviewer Ja4J [2/2]

(Q5) On Seed Variance: We repeated the main 86M experiment across three independent seeds (42, 43, 44), retraining both String Atlas and all five full specialists. The complete per-route results are reported below.

String Atlas (BPB ↓)

RouteSeed 42 (paper)Seed 43Seed 44Mean ± std. dev.
General1.098901.098101.102301.09977 ± 0.00223
Clean web1.098701.098001.102201.09963 ± 0.00225
Code skew1.106001.105201.109701.10697 ± 0.00240
Sci/math1.103801.103001.107301.10470 ± 0.00229
Wiki/QA1.106001.105201.109601.10693 ± 0.00234
Route Avg.1.102681.101901.106221.10360 ± 0.00230

Full Specialists (BPB ↓)

RouteSeed 42 (paper)Seed 43Seed 44Mean ± std. dev.
General1.088801.087401.088801.08833 ± 0.00081
Clean web1.082201.086201.088001.08547 ± 0.00297
Code skew1.090501.091601.092101.09140 ± 0.00082
Sci/math1.091101.092901.090101.09137 ± 0.00142
Wiki/QA1.092201.095101.094101.09380 ± 0.00147
Route Avg.1.088961.090641.090621.09007 ± 0.00096

BPB Gap: String Atlas − Full Specialists (↓)

RouteSeed 42Seed 43Seed 44Mean ± std. dev.
General+0.01010+0.01070+0.01350+0.01143 ± 0.00181
Clean web+0.01650+0.01180+0.01420+0.01417 ± 0.00235
Code skew+0.01550+0.01360+0.01760+0.01557 ± 0.00200
Sci/math+0.01270+0.01010+0.01720+0.01333 ± 0.00359
Wiki/QA+0.01380+0.01010+0.01550+0.01313 ± 0.00276
Route Avg.+0.01372+0.01126+0.01560+0.01353 ± 0.00218

The results are relatively stable across three seeds. The mean gap to full specialists is +0.01353 ± 0.00218 BPB, closely matching our original +0.01372 BPB. The near-specialist performance of String Atlas is not specific to the original training seed.

(W4) On Presentation and Related Work: Thank you for bringing this up. We will move the critical sections of the related-work discussion from Appendix A into the main text, and standardize terminology throughout the paper, particularly the use of tokenizer route, atlas row, and BPB gap.


Once again, we appreciate your thoughtful feedback and positive assessment of our work. If you have any other questions, please let us know.

Thank you for your detailed response. The seeded runs are much appreciated and underline the main finding that the gap to the specialists remains consistent across seeds. Thank you also for the per-benchmark and per-route downstream results, which demonstrate String Atlas's consistent performance across routes.

The routing experiments are interesting. However, the per-domain results show that the matching route is never the best in BPB. Therefore, in my opinion, the routing results demonstrate a token-count benefit rather than the quality benefit implied by the motivation.

This makes me wonder whether prefix-based compression is the right routing policy. I would be interested in a brief justification for this choice. Would a likelihood-based policy, which selects the route with the lowest BPB on the prefix, work better, at least for longer documents? This approach would target quality directly, but scoring the prefix once per route adds prefill costs, which could outweigh the token savings, e.g. for short documents. A full-document oracle that selects the best route for each document could show the maximum quality gain any routing policy could achieve.

Reviewer_SKK9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors of this paper introduces “String Atlas” an architecture for language modelling to support more than one tokenizers on the basis of the shared lexical parameters by byte strings. Authors evaluate the method across domain-specific, vocab-size, and with distinct tokenizers to show near equivalent performance with a lower training cost. They also experiment with model scales, downstream benchmarks, and other ablations to validate their approach.

Critical strengths
  1. Authors challenge the assumption that LMs must be tied to single tokenizer and tried to establish a potential alternative to training separate models or post-hoc adaptations.

  2. String Atlas near-match the performance across domain-specific, granularity-based, distinct tokenizers. The advantage persists across model scales, and perform well-motivated ablations and analysis.

Critical weaknesses
  1. Although I find the paper interesting, the paper doesn’t demonstrate when and how should we get practical benefits from multiple tokenizers. In my opinion, I find the practical benefits insufficiently established. Also, the proposed approach, String Atlas depends on vocab overlap but I don’t see the clear relationships of the overlaps with the performance, also it has limited downstream evals.

  2. I think the actual benefits could come from the demonstration of multilingual cases. The evaluation primarily on English-language pretraining data, although the authors consider domain variations, but it doesn’t evaluate multilingual tokenizers with differing writing systems etc.

Questions for authors

See the "Critical Weaknesses" as well.

  1. How does the “String Atlas” perform when we have less vocab overlaps or the num of supported tokenizers increases?

  2. The paper is motivated around multilingual fairness, but I could not find enough supporting evidences in the paper. Does the approach retain its advantages when the tokenizers are trained on different languages?

4 comments
AuthorsOpenReview ↗

Response to Reviewer SKK9

Thank you for the review. We have conducted additional experiments addressing your concerns, particularly regarding practical benefits, multilingual evaluation, and vocabulary overlap. The main results are presented in our General Response.

(W1) On Practical Benefits: We now demonstrate that String Atlas, paired with a simple shortest-prefix routing policy, can substantially reduce sequence lengths with minimal quality degradation. Across six text domains and six languages, we achieve 8.20% and 9.18% fewer tokens, respectively, with BPB increases of just +0.017 and +0.0234. See our General Response for the full results.

On Vocabulary Overlap and Performance: Our new multilingual experiments use seven independently trained tokenizers spanning English, Spanish, French, Arabic, Chinese, and Japanese. We find that when we introduce these different languages and writing systems, pairwise byte-string vocabulary Jaccard similarities drop to about 0.500 to 0.635, significantly lower overlap than the domain-specific tokenizers in the main paper results. Even with this variation and an increase to seven supported tokenizers, String Atlas remains within +0.0277 BPB of full-budget specialists. This is in stark contrast to the other compute-matched baselines. Private Tables and Compute-Matched Specialists are far behind, trailing by +0.4353 and +0.5180 BPB, respectively.

(W2) On Multilingual Evaluation: We have now conducted precisely this experiment. Our new multilingual suite includes six languages spanning multiple writing systems, with independently constructed language-specific tokenizers. The results confirm that String Atlas retains near-specialist quality while enabling practical compression gains through language-aware routing. See W1 and our General Response.

(Q1) On Decreasing Vocabulary Overlap and Increasing Tokenizer Count: String Atlas exploits a simple property of real-world tokenizers: even when their segmentation rules differ substantially, they often share a large core of frequently occurring byte strings. Sharing these rows pools supervision across routes, which is the mechanism demonstrated in Section 7.

We also tested more heavily specialized domain tokenizers in Appendix C. The atlas grows from 50.2k to 58.9k rows, yet the mean gap to full specialists increases only from +0.0137 to +0.0189 BPB. Our new multilingual experiments further extend the evaluation from five to seven tokenizers, maintaining a gap of just +0.0277 BPB. We expect these gaps to further reduce with longer training and larger scale models, in line with the trends we observe for the 500M and 1B-scale results.

To reiterate, String Atlas does not require identical tokenizers or compatible segmentation rules. It takes advantage of the lexical overlap that naturally exists between independently constructed tokenizers. Our experiments show that this remains effective across domains, vocabulary sizes, tokenization algorithms, and now multiple languages and writing systems.

(Q2) On Multilingual Fairness: See W1–W2 and our General Response. The new experiments demonstrate that a single model can support language-specialized tokenizers across six languages, reducing token counts by 7.48–13.16% relative to the general multilingual tokenizer while maintaining near-specialist language-modeling quality.


Thank you again for the constructive review; we would be glad to clarify anything further.

AuthorsOpenReview ↗

Following up, response would be appreciated.

Dear Reviewer SKK9,

We wanted to briefly follow up on our response above. We have now completed the multilingual experiments you suggested, along with additional routing and vocabulary-overlap evaluations.

If these new results adequately address your main concerns, we would be grateful if you could update your score accordingly. We would also be happy to clarify anything further before the discussion period ends.

I thank the authors for their responses. I still have reservations about whether the reported token savings translate into meaningful practical benefits for downstream performance, and broader multilingual applicability.

After considering the rebuttal and other reviewers' discussions, I lean toward maintaining my original rating.

AuthorsOpenReview ↗

Response to Reviewer SKK9

Thank you for following up. We may not have made the new evidence sufficiently clear, because the two reservations in your comment are precisely the two issues for which we added experiments.

First, the token savings are themselves a practical efficiency benefit: the model processes 8.20% fewer tokens across domains and 9.18% fewer tokens across languages, using an ordinary token-based Transformer and a router that was not trained with the model. We do not claim that shortening a sequence must increase multiple-choice accuracy. The downstream experiments ask the complementary question of whether supporting several tokenizer routes preserves task performance. Across seven benchmarks and three model scales, String Atlas has the best macro-average among all one-model methods at every scale; at 500M it even slightly exceeds the full-specialist average. String Atlas receives the pretraining compute of only one specialist, whereas supporting all five routes with separate specialists requires five complete pretraining runs. The per-route results show the same story: String Atlas is the strongest one-model method on all five routes at 500M, and on four of five routes at both 86M and 1B. In other words, the shorter sequences do not come from replacing the model with a weaker tokenizer interface: String Atlas retains near-specialist language-modeling and downstream quality while making the efficiency gain available.

Second, we added the multilingual experiment you requested. It trains one String Atlas model with seven independently constructed tokenizers spanning English, Spanish, French, Arabic, Chinese, and Japanese, including three writing systems that differ sharply from Latin script. Even with substantially lower vocabulary overlap, the model remains within 0.0277 BPB of seven separately trained specialists, while the other one-model baselines are more than 0.4 BPB behind. Every language-specific tokenizer shortens its corresponding language, with savings from 7.48% to 13.16%, and the simple router saves 9.18% on the balanced multilingual stream. This gives a direct positive answer to your original question of whether String Atlas retains its advantage across languages and writing systems.

We agree that evaluating more tasks and languages would always be useful, but we do not think exhaustive downstream or multilingual coverage is necessary for the claim of this paper. The original paper asks whether one model can natively support several independently constructed tokenizers at near-specialist quality, and it already establishes this across domains, vocabulary sizes, tokenization algorithms, model scales, and downstream benchmarks. During rebuttal, we went beyond that scope and showed that even two simple post-hoc routers already yield useful compression, and that the approach continues to work across six languages and multiple writing systems.

If there is a specific result you still believe is missing, we would sincerely appreciate knowing what it is. Given that the revised paper now directly addresses both practical routing and the multilingual setting raised in your review, we respectfully ask that you reconsider your assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of additions in the revised PDF

We thank the reviewers again for their time and thoughtful discussions. We have uploaded a revised PDF that incorporates the new experiments and clarifications from our responses. All new material is highlighted in blue. For convenience, we summarize the changes below and indicate which reviewer question each one addresses.

  • Simple routing and practical motivation --- Reviewer xssh (W1/Q1), Reviewer Ja4J (W2 and routing follow-up), and Reviewer SKK9 (W1). We added a main-text section and a full appendix evaluation of two simple routers. Shortest-prefix routing uses the first 512 bytes of a document and reduces the token count by 8.20%. A likelihood-based router improves over always using the General route on both axes, reaching lower BPB while using 3.74% fewer tokens. We also include a continuation oracle to show the remaining headroom for a learned router.

  • The complete cost relative to a single-tokenizer model --- Reviewer xssh (General-specialist follow-up) and Reviewer Ja4J (follow-up point 3). We evaluated the separately trained General specialist on the same documents and continuation bytes. Learning five tokenizer interfaces costs 0.0112 BPB, and shortest-prefix routing adds 0.0171 BPB. The full gap is 0.0283 BPB, so the model retains all five interfaces and obtains 8.20% shorter sequences while remaining within 0.03 BPB of the specialist.

  • Native Union-Longest comparison --- Reviewer xssh (Q2 and Union-Longest follow-up). We added native Union-Longest to both the mixed-domain routing table and the full compression matrix. It saves 7.11% of the tokens at 1.3827 BPB, while shortest-prefix String Atlas saves 8.20% at 1.3964 BPB. Union-Longest has slightly lower loss, but it does not dominate String Atlas on compression; String Atlas produces the shorter sequence, preserves the five original tokenizer interfaces, and uses only the selected 32k softmax.

  • Compression on the expanded corpus --- Reviewer Ja4J (follow-up point 4). We now state clearly that Table 1 used 2 MiB per domain, whereas the routing study uses 5 GiB per domain. The appendix reports the complete 5 GiB compression matrix and explains the change in document distribution between the two evaluations.

  • Multilingual pretraining and routing --- Reviewer SKK9 (W2/Q1/Q2) and Reviewer xssh (practical motivation). We added a new experiment with seven independently trained tokenizers spanning six languages and several writing systems. One String Atlas model remains within 0.0277 BPB of seven full specialists, reducing the seven tokenizer-local tables from 224k entries to about 73k shared rows. The same simple shortest-prefix router then reduces the multilingual token count by 9.18%.

  • Per-benchmark and per-route downstream results --- Reviewer Ja4J (Q3) and Reviewer SKK9 (limited downstream evaluation). The revised appendix now gives the full downstream breakdown rather than only the aggregate. String Atlas has the best macro-average among the one-model methods at all three scales, and its performance remains consistent across tokenizer routes rather than depending on one favorable route.

  • Related work and presentation --- Reviewer Ja4J (W4). We moved the relevant comparison with byte-level, multilingual, and modular-tokenizer methods into the main text and standardized the terminology used for tokenizer routes, atlas rows, and BPB gaps.

Our original submission examined whether one model could natively support several independently constructed tokenizers while remaining close to specialist quality, and our main experiments already show that. The revision now goes a step further. We show that simply trying two post-hoc routing policies is already enough to produce useful compression and quality improvements across domains and languages. These routers are intentionally simple and were never trained with the model, leaving joint training of the router and language model as a natural next step.

We hope the revised PDF makes both the scope of the paper and the practical value of multi-tokenizer pretraining clearer. We would be grateful if the reviewers would consider the new results in their final assessments, and we are happy to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.