- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how tokenization and training objectives jointly affect the scaling behavior of overtrained language models. It introduces two methods for converting a token-based teacher’s predictions into byte-level distillation targets: Marginalize-It, and End-Of-Token (EOT). The paper evaluates eight tasks in Llama 3-8B. The central empirical observation is that token models perform better at lower training budgets, while byte models improve with additional training and approach their performance.
Critical strengths
- The research question is relevant and the experimental comparison is substantial.
- Explicit token termination is a simple and potentially useful construction.
- The distinction between validation BPB and downstream performance is informative.
Critical weaknesses
- The observed and extrapolated results should be distinguished more prominently. At the largest reported training scales, Table 17 reports average accuracy of 44.6% for EOT distillation and 44.4% for token distillation. The widely emphasized 52.4% versus 48.4% comparison is an asymptotic prediction, not an observed four-percentage-point improvement (Tables 4 and 14).
- The data and storage savings use potentially misleading comparison criteria. The 6.35× data reduction in Table 6 follows from equal-compute accounting and EOT’s higher computational cost per unit of text, rather than measured data requirements at matched performance. 3.The comparisons do not isolate the source of improvement or establish deployment efficiency. The settings differ in target truncation, qk-norm, batching, and boundary information, making EOT’s gains difficult to attribute to a specific mechanism. Fewer parameters also do not establish lower deployment costs, and matched inference-cost and end-to-end distillation-cost measurements are missing.
Questions for authors
none
We thank the reviewer for their valuable suggestions!
We are glad that they find our research question relevant and experiments substantial, our method useful and the distinction between validation BPB and downstream performance is informative. We provide below the response to each of their questions.
Q1: Distinguishing Observed and extrapolated results
Thank you for the suggestion. We totally agree that this distinction should be more clear. Specifically, in both Table 4 and Table 14 we changed the column title to “predicted a’ (%)”. In Table 7 as well we changed the column header to predicted Asymptotic Avg. Accuracy. (%). We also updated the captions of Tables 3, 4, 5, 6 to make this distinction more clear.
Q2: Data/storage savings analysis at matched performance.
Thank you for bringing this up!
Yes, our current analysis is a prediction from the scaling law calculated at the compute budget when the distilled End-Of-Token-1B will surpass Token-1B.
However it is still possible to do data efficiency analysis for our models in the trained range. Based on this suggestion we plot task performance vs training data in bytes graphs for our trained range in appendix C Data efficiency Analysis. At matched performance compared to token distillation baseline, the End-Of-Token 1B model uses less data (see dashed line). We hypothesize that this gap will widen with additional data and compute as token models will saturate compared to byte models.
Q3: Isolating the EOT method's source of improvements
Thank you for this point!
For both Bytes-1B and End-Of-Token-1B, we use a sequence length such that we preserve the same context as that of the Token-1B model. In Table 7, we provide the exact values of the sequence lengths. For token it is 2048, for Bytes-1B it is 2048 * 4.5=9216 (considering the compression rate of 4.5) , 2048 * 5.5= 11264 (accounting for the additional <eot> token every 4.5 bytes).
Intuitively, sequence length truncation/ batching should only hurt performance as they reduce the context/ reduce the amount of effective text used for each optimization step compared to token models. However, we observe that they still match distilled token models at higher compute regimes, so it seems like these factors do not have major impact on the task performance of byte models.
The Data Efficiency Analysis in appendix C also suggests that End-of-Token-1B appears to be the overall winner in the trained range for matched data in bytes – an effect that could be attributed to the exactness as well as the additional compute spent on modeling the data.
Q4: What are the implications on inference costs?
Thank you for this point! We have also included this explanation in our common response.
This work is just the first step towards understanding the scaling behavior when two axes – the training objective and the tokenization scheme are varied simultaneously. The byte models still remain significantly expensive at inference and we have mentioned this as a limitation in the discussion section.
By training/distilling models by varying the number of parameters, tokenizers with different compression rates (ours has 4.5), architectures (e.g. MoEs) simultaneously could reveal more insights about which configurations are optimal for a given training compute budget and inference cost constraints. We have left this investigation as future work.
If there are any remaining questions, we would be happy to answer!