- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
As its title suggests, this paper studies the scaling behavior, learned representations, and information allocation of byte language models. The authors pretrain a series of byte transformers ranging from 200 M to 3 B parameters and fit Chinchilla-style scaling laws. Here, they find that byte models outperform their subword counterparts at fixed parameter counts, but not with matched training compute. Byte transformers demonstrate further improvement on a mixture of synthetic character manipulation tasks and common LLM and OCR benchmarks. Subsequently, the authors perform systematic ablations, finding that their byte transformers develop abstractions that loosely resemble an implicit tokenizer. Finally, the authors characterize the distribution of per-byte prediction difficulty and exploit the non-uniform structure for speculative decoding.
Critical strengths
Byte transformers outperforming subword transformers on the reported benchmarks is genuinely interesting. My initial intuition would have been that increasing the number of predictions to generate the same text might accumulate more error. I’m curious as to why this isn’t the case, and find these results useful for the community
Critical weaknesses
Because the underlying architecture and training recipes are different, one cannot wholeheartedly attribute the observed effects to the choice of bytes over tokens.
- In particular, the reported byte transformers use longer GDN conv kernels, longer n-gram hash embeddings, and token-superposition training. While the authors ablate on these choices for training dynamics/scaling results, the mechanistic results are only evaluated over the full model.
- How can we know that, for example, the formation of text abstractions actually occurs organically, and not because of the injected n-gram hash embeddings
- The looped-subword experiments suggest that allocating additional compute towards subword models can recover the initial fixed parameter scaling gap. As such, why should one prefer byte-level language modeling beyond providing just another scaling axis? -- I know the point of this paper isn’t just to show capabilities, but I feel the authors should be forthcoming about this limitation, and further stress-test the result before including it as a main finding. More on this below
Depth > Breadth. The paper reads as if the authors pretrained a collection of models increasing in scale, performed several somewhat related experiments, and simply aggregated the findings into the paper. While admittedly, there does exist somewhat of a narrative around byte-level computation being useful as a whole, each subsection feels duly half-baked. Here are a few examples
- Scaling -- See above points
- Implicit tokenization -- The attention analysis is heuristic and performed on a small synthetic copying task. Further, instead of using representations identified from this analysis, the boundaries from frozen-layer experiments are determined by a separate model. At this point, it becomes unclear what the ensuing experiments are trying to accomplish, or more broadly, whether they actually validate the mechanism being described. The authors should consider tightening this analysis, making it more rigorous and systematic
- Information allocation -- The spec decoding results are preliminary/proof of concept at best. First, they occur in open loop (oracle simulation), rather than actual decoding. The authors report 3.4x more accepted tokens, but neglect that the accepted number of bytes per target forward pass is almost identical to the subword counterpart. Again, the authors should be more forthcoming about both limitations here instead of sweeping them under the rug and hoping a reviewer doesn’t find them.
This paper, and the community I suppose, would greatly benefit from choosing one or two of the findings and understanding them with much greater depth.
Questions for authors
I’m reasonably concerned about whether or not the amount of source text is held fixed between the byte and subword regimes. Could the authors clarify whether their scaling comparisons are fair despite the TST exposure multiplier?
3 comments
Response to Reviewer badS (Part 2)
W2.1. Central storyline and implications
Our central perspective is that byte-level modeling turns longer sequences into a useful scaling axis. Meanwhile, BLT and H-Net introduce local structure and improve efficiency through architectural changes, whereas we show that a flat Transformer can also learn useful local structures and offer opportunities for efficient inference.
In Sec. 2, we examine token granularity as a direction for allocating computation alongside model size and recurrent depth. The compute-optimal frontiers show that BYTE achieves lower BPB at the same optimal parameter count with more training compute, while SUBWORD remains better at the same compute budget. Together with the fine-grained downstream gains and the orthogonal scaling directions explored through Loop/MoE, these results establish the value and trade-offs of sequence-length scaling.
Beyond these performance gains, Sec. 3 examines whether the local structures explicitly introduced by BLT and H-Net can also emerge within a flat Transformer. The copying probe reveals repeated access to locally aggregated information, while the frozen layer intervention tests its computational usefulness: BYTE retains substantial downstream performance when selected layers operate on compressed representations. Together, these results demonstrate that a flat Transformer can also learn useful local structures that support subsequent computation. Furthermore, as shown in Sec. 3.2 and Figure 4, BYTE retains more performance when the compressed stage is placed later in the network. This suggests that more byte-resolution processing may be useful before compression, motivating further exploration of LGL layer allocation.
Nevertheless, fine-grained byte modeling still incurs higher inference costs. Sec. 4.1 therefore compares BPB distributions across granularities, showing that byte modeling distributes the same source entropy more unevenly, concentrating information on key bytes and offering finer control over computation allocation. This motivates the drafts in Sec. 4.2 to handle low-information continuations. The resulting comparable accepted text per simulated verification provides a proof of concept for speculative decoding of BYTE to reduce inference overhead.
Together, our findings establish sequence length as a useful scaling axis and show that a flat Transformer can also develop computationally useful local structures. Moreover, the drafting results offer a path toward reducing inference overhead, supporting a simple approach that retains the flat architecture while exploiting the benefits of byte-level modeling.
W2.3. Scope and limitations of the speculative-decoding experiment
We agree with the reviewer that this experiment is a proof of concept: the 3.4× result refers to accepted tokens on reference-text prefixes, and establishing practical speedups requires end-to-end evaluation on generated histories, accounting for drafting and verification costs. We already discuss these limitations in G.2 (Implications for speculative decoding), but space constraints led us to keep this discussion in the appendix. We recognize that these limitations should be stated more clearly and we have now stated these points explicitly in the Limitations section and clarified them in Sec. 4.2.
Our experiment uses the simplest setting with existing small models as drafts. Developing an optimized speculative-decoding system would require additional design and specialized draft training, which is beyond the scope of this paper. Our aim here is to test whether drafts can handle predictable byte continuations. As shown in Table 3, BYTE and SUBWORD achieve comparable accepted bytes per simulated target forward pass, suggesting that BYTE’s higher acceptance can compensate for its finer token granularity.
Thank you for your response. I will increase my score to 2.
Response to Reviewer badS (Part 1)
We thank the reviewer for the careful assessment and constructive suggestions, particularly regarding recipe attribution and the connection between our mechanistic experiments.
W1.1, Q1. Recipe differences and source-text exposure
Please refer to our General response for the recipe motivation and symmetric TST/hash comparisons. In Sec. 2.2, the IsoFLOPs sweep compares models under fixed training FLOPs, with the fits explicitly accounting for source-byte exposure, including TST’s multiplier.
For the main model suite, BYTE receives approximately 1.67× the source-byte exposure of raw SUBWORD. We will clarify Footnote 3: approximate exposure matching applies to ordinary next-token training, as stated in App. B.1.
Additionally, we note that the convolution widths and hash configurations provide comparable local text spans that account for the different input granularities: during ordinary next-token training and inference, BYTE’s conv size of 16 and SUBWORD’s conv size of 4 correspond to approximately 16 and 18 bytes, respectively. Since n-gram lengths are also measured in each family’s token units, using the same n would not match their text spans. BYTE uses n = 5–8 with four tables of 32,768 slots each, while SUBWORD uses n = 2 with one table of 131,072 slots. Thus, both families use the same total number of hash slots and matched text spans.
W1.2, W2.2. Evidence for useful local abstractions
To examine the role of TST and hash directly, we repeated the Figure 4 intervention on raw BYTE.
Although performance retention is lower than with TST+hash, the overall trend remains similar: compression in later layers preserves substantial performance and outperforms random segmentation at the same compression ratio. Importantly, this supports the emergence of useful local structures under ordinary BYTE training, while TST and hash further strengthen their effectiveness.
Regarding the connection between the experiments, the copying probe identifies attention patterns consistent with local aggregation, while the frozen intervention tests whether aggregated representations can support subsequent computation. Boundaries obtained from the smaller model remain useful for the larger target, suggesting transferability of local partitions across scales. Furthermore, retaining downstream performance with all parameters frozen provides functional evidence beyond the attention patterns themselves.