acceptodds
Under review as a conference paper at ICLR 2027

Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models

Abstract

Small models are made more capable through distillation from a larger one that shares their tokenization scheme. However, do distilled byte and token models behave similarly in terms of scaling trends as compute and data increases? To enable this comparison, we introduce two variants to efficiently convert token logits to Byte Logits: 1) approximate: Marginalize-It , and 2) exact: End-Of-Token. We then present the first large scale study of overtraining decoder-only dense transformer models varying two dimensions simultaneously: the tokenization scheme ( Tokens , Bytes , Bytes w/ <eot> ) and the training objective (Distillation vs. Cross-Entropy), sweeping layer-parameter-matched models with ≈ 1 billion parameters up to 1 trillion bytes of data. Across eight benchmarks spanning three categories: Multiple Choice QA, Language Generation, and Machine Translation, we find that Token-1B models outperform byte models ( End-Of-Token-1B and Bytes-1B ) in the low-FLOP regime but eventually plateau; byte models start worse yet surpass Token-1B models with more compute, reaching a higher downstream task performance ceiling. Extrapolating the average top-1 error vs. validation BPB scaling laws predicts that, asymptotically, distilled End-Of-Token-1B outperforms distilled Token-1B by up to 4% on averaged downstream task performance. They are also far more data efficient, matching the performance of distilled Token-1B using only one-sixth of the training data. Moreover, by operating over a small vocabulary of ≈ 256 bytes instead of ≈ 100K tokens, they circumvent the need for top-k truncation during logit dumping, while also reducing logit storage costs to roughly one-fifth. Finally, our downstream performance scaling laws predict that our distilled End-Of-Token-1B models asymptotically surpass the Llama 3.2-1B (Meta AI, 2024), Gemma-3-1B-pt (Kamath et al., 2025), and Gemma 2B (Team et al., 2024) models on averaged downstream tasks by up to 6.5%, 8.1%, and 2.1%, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Distillation, Scaling, Overtraining, Tokenization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_zizEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how tokenization and training objectives jointly affect the scaling behavior of overtrained language models. It introduces two methods for converting a token-based teacher’s predictions into byte-level distillation targets: Marginalize-It, and End-Of-Token (EOT). The paper evaluates eight tasks in Llama 3-8B. The central empirical observation is that token models perform better at lower training budgets, while byte models improve with additional training and approach their performance.

Critical strengths
  1. The research question is relevant and the experimental comparison is substantial.
  2. Explicit token termination is a simple and potentially useful construction.
  3. The distinction between validation BPB and downstream performance is informative.
Critical weaknesses
  1. The observed and extrapolated results should be distinguished more prominently. At the largest reported training scales, Table 17 reports average accuracy of 44.6% for EOT distillation and 44.4% for token distillation. The widely emphasized 52.4% versus 48.4% comparison is an asymptotic prediction, not an observed four-percentage-point improvement (Tables 4 and 14).
  2. The data and storage savings use potentially misleading comparison criteria. The 6.35× data reduction in Table 6 follows from equal-compute accounting and EOT’s higher computational cost per unit of text, rather than measured data requirements at matched performance. 3.The comparisons do not isolate the source of improvement or establish deployment efficiency. The settings differ in target truncation, qk-norm, batching, and boundary information, making EOT’s gains difficult to attribute to a specific mechanism. Fewer parameters also do not establish lower deployment costs, and matched inference-cost and end-to-end distillation-cost measurements are missing.
Questions for authors

none

1 comment
AuthorsOpenReview ↗

We thank the reviewer for their valuable suggestions!

We are glad that they find our research question relevant and experiments substantial, our method useful and the distinction between validation BPB and downstream performance is informative. We provide below the response to each of their questions.

Q1: Distinguishing Observed and extrapolated results

Thank you for the suggestion. We totally agree that this distinction should be more clear. Specifically, in both Table 4 and Table 14 we changed the column title to “predicted a’ (%)”. In Table 7 as well we changed the column header to predicted Asymptotic Avg. Accuracy. (%). We also updated the captions of Tables 3, 4, 5, 6 to make this distinction more clear.

Q2: Data/storage savings analysis at matched performance.

Thank you for bringing this up!

  • Yes, our current analysis is a prediction from the scaling law calculated at the compute budget when the distilled End-Of-Token-1B will surpass Token-1B.

  • However it is still possible to do data efficiency analysis for our models in the trained range. Based on this suggestion we plot task performance vs training data in bytes graphs for our trained range in appendix C Data efficiency Analysis. At matched performance compared to token distillation baseline, the End-Of-Token 1B model uses less data (see dashed line). We hypothesize that this gap will widen with additional data and compute as token models will saturate compared to byte models.

Q3: Isolating the EOT method's source of improvements

Thank you for this point!

  • For both Bytes-1B and End-Of-Token-1B, we use a sequence length such that we preserve the same context as that of the Token-1B model. In Table 7, we provide the exact values of the sequence lengths. For token it is 2048, for Bytes-1B it is 2048 * 4.5=9216 (considering the compression rate of 4.5) , 2048 * 5.5= 11264 (accounting for the additional <eot> token every 4.5 bytes).

  • Intuitively, sequence length truncation/ batching should only hurt performance as they reduce the context/ reduce the amount of effective text used for each optimization step compared to token models. However, we observe that they still match distilled token models at higher compute regimes, so it seems like these factors do not have major impact on the task performance of byte models.

  • The Data Efficiency Analysis in appendix C also suggests that End-of-Token-1B appears to be the overall winner in the trained range for matched data in bytes – an effect that could be attributed to the exactness as well as the additional compute spent on modeling the data.

Q4: What are the implications on inference costs?

Thank you for this point! We have also included this explanation in our common response.

  • This work is just the first step towards understanding the scaling behavior when two axes – the training objective and the tokenization scheme are varied simultaneously. The byte models still remain significantly expensive at inference and we have mentioned this as a limitation in the discussion section.

  • By training/distilling models by varying the number of parameters, tokenizers with different compression rates (ours has 4.5), architectures (e.g. MoEs) simultaneously could reveal more insights about which configurations are optimal for a given training compute budget and inference cost constraints. We have left this investigation as future work.

If there are any remaining questions, we would be happy to answer!

Reviewer_NNpHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is a scaling study of training byte-level language models of around 1 billion parameters, compared with token-level models. The authors train students from scratch along two axes: the input format (tokens, raw bytes, or bytes with an end-of-token marker) and the training objective (plain cross-entropy or distillation from Llama 3-8B).

To distill a token teacher into a byte student, the authors propose two ways to turn token probabilities into byte probabilities. Marginalize-It sums the probabilities of the tokens that match the bytes so far and renormalizes. End-Of-Token adds an <eot> marker after every token so no probability is lost, which is exact.

The authors fit power laws of validation bits per byte (BPB) against training FLOPs, fit downstream error as a function of BPB, and extrapolate both to infinite compute. In the trained runs, token models are better at low compute and byte models catch up at the largest runs. From the extrapolations, the authors predict that distilled End-Of-Token models eventually beat distilled token models by up to 4 points of average accuracy, need one-sixth of the training data, and surpass Llama 3.2-1B, Gemma-3-1B and Gemma 2B. They also show that a lower BPB does not always mean better benchmark accuracy across tokenizations.

Critical strengths
  • Marginalize-It is a simple single-pass way to distill a token teacher into a byte student, and it helps at the scales trained (43.2 vs 41.8 average accuracy for plain bytes, Table 17). This matters because this finding is useful and do not depend on extrapolation.
Critical weaknesses
  • The main claims rest on extrapolation that the trained runs do not support:
    • At the same compute, token models are better or equal in the trained range, and byte models only catch up at the largest runs.
    • The predicted crossovers are 5.6× (End-Of-Token) to 24× (Marginalize-It) beyond the largest runs (Table 5).
    • The comparison with Llama 3.2-1B, Gemma-3-1B and Gemma 2B also uses extrapolated values: the trained End-Of-Token model scores 44.6, below Llama 3.2-1B (45.9) and Gemma 2B (50.3).
    • The fits have no uncertainty estimates and are never checked against a held-out larger run.
    • 'Smaller, stronger' ignores inference cost: per amount of text, End-Of-Token needs 6.35× the compute of the token model (Table 2).

This matters because the abstract and title present predicted results as findings, while the trained runs favor token models at equal compute.


  • The paper is hard to read and the setup is unclear:
    • Heavy formatting in the abstract and text, tables of raw fitted coefficients, and plots of 1-accuracy against descending BPB make the results hard to follow.
    • The training data is 'the Llama-2 (Meta AI, 2024) training mixture' (L151), cited to the Llama 3.2 blog, and the Llama 2 pretraining mixture in the appendix (L895). That mixture was never released, and the two validation sets are never described.
    • The related work lists papers without positioning the method against the closest token-to-byte distillation work.

This matters because it makes the results hard to assess or reproduce.

Questions for authors
  • [Clarification] The AI use statement mentions AI was used for brainstorming this work L476-477 and then again claim it was not use for brainstorming the work L482-483 can you clarify ?

  • [Suggestion] Can you validate the extrapolation, for example by fitting on the smaller scales and predicting your largest runs? Can you report uncertainty for the BPB limits and the predicted accuracies?

  • [Clarification] What exactly are the training data (L151 vs L895) and the two validation sets? Does BPB for End-Of-Token count the <eot> predictions?

3 comments
AuthorsOpenReview ↗

We thank the reviewer for taking the time to review our paper and provide thorough feedback.

We are glad that they find our single pass Marginalize-It method interesting and useful. We provide below the responses to the questions raised by the reviewer:

Q1: Observed Results vs Extrapolations

  • We agree that they should be distinguished clearly! As also noted in the paper summary, we use the word “predict” to make this distinction clear. Training these models longer to demonstrate the crossover point and the gains is indeed important which we have left as future work.

  • What is useful to the community is our analysis framework that allows us to make these predictions in the first place. Prior work for instance does not show why the same BPB value can mean different task performance and also why lower BPB is not an indicator of a better language model. Next prior work has shown that the task performance vs validation loss scaling laws exist, however we show that the tokenization scheme also affects them.

Q1A: Byte models catch up at the longest runs

That’s true!

  • What’s actually interesting is that byte models improve at a steeper rate with compute despite lagging intially which no prior work has shown yet. Our scaling law framework allows us to compare, quantify, and even predict the language modeling capabilities in terms of the benchmark performance for different tokenization schemes and training objectives – this has not been done in prior work.

  • We hypothesize that the differences in the steepness of the improvement is due to the differences in the amount of compute spent on a fixed amount of data when the context is kept equal. For example, as shown in Table 2, byte models spend much more compute on text worth 1 BPE token. Showing that the actual crossover exists is an important future direction. We have updated the abstract and the paper to use the word ‘predict’ to make this distinction more prominent and moved the predicted improvements using scaling laws into brackets in the abstract.

Q1B: Does the predicted crossover lie beyond the largest runs?

That’s true! We have added this as a limitation in the discussion section by adding the following sentence : “We leave the experimental validation of these predictions as future work.” If there are any specific instances in the paper where you think this distinction should be made more clear, we are happy to do so.

Q1C: Rationale for comparing asymptotic predictions with open-weight models

We agree that these are predictions! What's exciting about our work is that our scaling law analysis allows us to make these predictions in the first place, especially when the two models do not share a tokenizer. This was not shown in any prior work. We have updated Table 7 column with Predicted Asymptotic Avg. Accuracy(%) to emphasize that these are the predictions from the scaling laws. In Table 17 we show that the distilled End-Of-Token Distilled models already have surpassed the Gemma-3-1B checkpoint in the trained range.

Q1D: Scaling Law Sensitivity Analysis and Validating the predictions on a held out run

  • In appendix F we have provided the scaling law sensitivity (the value of asymptotic bits per byte: c) and also the quality of the fit in Table 12. We have left the experimental validation of our predictions as future work that’s exactly why we use the word 'predict' wherever needed!

  • We believe that overall trends (such as the amount of compute spent on a fixed amount of data when the context is kept the same affecting the scaling behaviors), the analysis procedure (1. validation BPB vs training FLOPs scaling trends, 2. averaged benchmark performance vs training flops and 3. averaged benchmark performance vs validation BPB) we use for enabling fair comparisons for different tokenization schemes and training objectives will be valuable to the scientific community.

  • In addition we have also added a section (Appendix I: validating the scaling law predictions)

Q1E: What are the implications on inference costs?

Thank you for pointing it out! The byte models are certainly expensive at inference time. Future work should indeed seek ways to make them more efficient. We have clearly mentioned this as a limitation in the Discussion section. Our work is just the first step towards thinking in that direction! We have also included this point in the common response.

Q2 Readability and Reproducibility

Thank you for the feedback! Regarding reproducibility: We think that the overall scaling trends should be reproducible for 1B parameter models. Future work should certainly consider training models with different model sizes, datasets and evaluating on more benchmarks.

Q2A: Formatting and Tables with Equations

To improve readability we have made a number of changes.

  • Text formatting: We had used colors to highlight keywords to make it easier to map the experiment with the figures. However as per the suggestion we have decided to remove the colors and text formatting.

  • Tables of raw fitted coefficients: We have decided to move the tables with raw fitted coefficients to appendix.

  • plots of (1-accuracy) against descending BPB: This was a deliberate decision as these plots are derived from prior work that uses the same protocol (Average top-1 error vs validation loss). What is novel about our work and missing from the existing literature is that, we show that these trends also exist and vary for different training objectives and tokenization schemes when we change the x axis to Bits per byte instead of the validation loss keeping the underlying data exactly the same. We show that BPB (which is regarded as a tokenizer independent metric in the community for fairly evaluating language models) is, in fact, not a tokenizer independent metric in the context of benchmark evaluations.

Q2B: Clarification on the training dataset and validation datasets.

Thank you for this question! That was a mistake while editing the paper.

  • We use Llama 2 pretraining mixture for our experiments. We have updated the paper (L150) with this change.
  • Regarding validation sets: we use two internal datasets to plot the power laws. However we don’t think the procedure we use for scaling trends I, II, III, to demonstrate the benchmark performance vs validation BPB discrepancy and the overall conjecture that the language model scaling behavior depends on the amount of compute spent on a fixed amount of data (when context is kept the same), should change with changes in the validation data. We have added a sentence in the paper to clarify that the validation datasets are internal in Section 3 and Appendix A.2.

Q2C: Additional discussion on Token to Byte distillation methods

Thank you for the suggestion! We have added a discussion of related work on token to byte distillation and compared these approaches with ours to clarify the distinctions.

Q3: Clarification regarding the AI usage statement

Thank you for pointing this out! We used an AI tool to provide additional intuition and to brainstorm the scenario in Table 23, where the same BPB can lead to different accuracies. However, we did not use AI tools to brainstorm the experiments or main ideas. We have revised that sentence to make this distinction clear.

Q4: Suggestions on Validating the scaling law extrapolations

Thank you for this suggestion! We have added additional analysis in the appendix I Validating the scaling law predictions (Table 22, Table 23.). Specifically we compare the predicted values (BPB and averaged accuracy) on a held out largest training run with the observed values as that scale as well as the asymptotic predictions for our lr-4e-3 experiment. We conclude that the main results i.e. the prediction that the distilled End-Of-Token-1B will surpass distilled Token-1B remains consistent when we omit the highest compute points from each experiment.

Q5: Clarification on Validation datasets and BPB calculation for End-Of-Token

Great questions!

  • The two datasets are internal datasets: one of them is not related to the pretraining dataset while the second dataset is a held out subset from the llama 2 training mixture. Note that the overall benchmark performance vs Training FLOPs plots (Scaling Trends II) do not depend on the validation dataset. We use the validation datasets to 1) demonstrate that the lower bits per bytes does not necessarily mean a better language model and it is not a tokenizer independent metric which is a common misconception 2) the task performance is a better measure for comparing models with different tokenization schemes and training objectives. 3) show that it is possible predict the quality of a language model by using downstream task performance vs validation bpb scaling laws 4) show that considering multiple validation sets for making the final predictions could be valuable.

  • Yes we do include <eot> predictions in the BPB calculations. Intuitively, they may have more contributions in the BPB calculation initially during training, however, as the training progresses, the language model would learn the BPE tokenization implicitly (as it is an easier task) by assigning the probability of 1 to the <eot> tokens and hence the eot tokens themselves won't eventually contribute much into the BPB.

We thank the reviewer again, for a detailed feedback. If there are any additional questions, we would be happy to answer!

Reviewer_JY38OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studied if token or byte is a better solution for large-to-small model distilaltion. It proposed some specitial tokens to enhence the idea of byte distillation and did study on llama-3 and gemma-3 series.

Critical strengths

the paper showed a interesting and debating idea and used a scientific way to address the problem.

Critical weaknesses

Weakness: (1) study are performed on two very old model families. (2) The scale is probably not enough to support the cliam. (3) the inference cost of byte token is unaccetaple.

Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

We thank the reviewer for taking the time to review our paper and are glad that they find the idea interesting

We provide below the response to the questions raised by the reviewer:

Q1: Clarification on Model families used for comparison

In our work, we primarily use Llama style dense transformer models trained on Llama-2 data mixture. We compare our predicted performance with open sourced Llama and Gemma checkpoints of comparable sizes as pretrained models are readily available on huggingface.

Q2: At what scale does the crossover appear?

Thank you for pointing this out! Training for more tokens is indeed an interesting future direction to demonstrate the crossover point! We tried to clarify this point in the common response.

Q3: What are the implications on inference costs?

Thank you for bringing this up! The inference costs of our byte models are indeed high and we mention this clearly in the Discussion section. Our main contribution is that, through the lens of scaling laws, we predict that distilling into bytes is asymptotically better than distilling into token models for 1B parameter (without thinking about deployment or inference costs). Improving the inference costs of byte models without losing model performance is indeed an exciting direction for future work.

If there are any additional questions, we would be happy to answer!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you to the reviewers for valuable feedback!

We are grateful to the reviewers for taking the time to review our work and providing valuable feedback.

We are glad that the reviewers find the idea interesting (Reviewer JY38), our methodology simple (Reviewer NNpH, Reviewer zizE), the experimental comparisons substantial, and the distinction between validation BPB and downstream performance informative (Reviewer zizE).

We provide below our response to some of the common questions raised by multiple reviewers. If there are any additional questions and comments, we would be happy to discuss them further.

[Reviewer JY38, Reviewer NNpH, Reviewer zizE] Observed vs Extrapolated results

We agree that training each of our experiments for more tokens until the crossover is indeed important, if expensive. This is exactly why we use the word ‘predicted’ in our abstract and we have left this investigation for future work.

  • We have added this point in the discussion section as a limitation by adding the following sentence – “We leave the experimental validation of these predictions as future work” and used the word predicted in abstract as well as Tables to make this clear to the reader.

  • Overall, what is more useful to the community is that our scaling law analysis allows us to make these predictions in the first place, which no prior work has done. First, we demonstrate that using bits per byte as a metric to determine the language modeling capability can be misleading. We show that two tokenization schemes can have exactly the same BPB but wildly different benchmark performance and that the lower BPB does not necessarily mean a better language model.

  • This is important because many works in the tokenization community use BPB as an evaluation metric to compare two language models with different tokenization schemes as well as for model selection. We show that this can lead to inconsistent conclusions and hence averaged benchmark performance vs BPB scaling laws are more useful to make scaling decisions.

[Reviewer JY38, Reviewer NNpH, Reviewer zizE] Inference costs

It is true that the inference costs are high for the byte models. We have already mentioned this in the Discussion section: “Comparing the asymptotic performance of a Token model against that of a End-Of-Token transformers with matched inference cost could reveal whether distilled End-Of-Token-1B transformers are also efficient at inference time. This analysis is beyond the scope of this study.” This work is just the first step towards understanding the scaling behavior when two axes – the training objective and the tokenization scheme are varied simultaneously and opens many exciting avenues for future work!

[Reviewer JY38, Reviewer NNpH, Reviewer zizE] Overall readability

We had initially used formatting and colors to help readers map the experiments with the figures. However based on the feedback from the reviewers we have decided to remove the formatting. We have also made changes in the paper to distinguish between the predicted values and the results observed in the trained regime more prominently in the abstract as well as the Tables 3, 4, 5, 6. If there are any additional suggestions that will improve readability, we would be happy to incorporate them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.