acceptodds
Under review as a conference paper at ICLR 2027

Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation

Abstract

Tokenizer-free language models remove the inductive bias of fixed tokenizers by modeling text directly as bytes, but the resulting longer sequences substantially increase computation and eliminate explicit text abstractions. We ask whether this additional computation can be useful, and whether standard Transformers can learn the abstractions that tokenization provides. We study these questions on Transformers without specialized tokenization-related architectures. With token-superposition training and hash embeddings, byte Transformers consistently outperform subword Transformers as model size scales. We further find that byte Transformers build local text abstractions as external tokenizers: a set of segmentation-like positions are used to collect local context representations, and restricting up to of intermediate layers to these local representations preserves downstream performance. Finally, these learned structures induce highly non-uniform generation difficulty, with uncertainty concentrated near local structure boundaries; exploiting them for speculative decoding yields more accepted tokens than in subword Transformers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
byte language model, scaling law

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_badSOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

As its title suggests, this paper studies the scaling behavior, learned representations, and information allocation of byte language models. The authors pretrain a series of byte transformers ranging from 200 M to 3 B parameters and fit Chinchilla-style scaling laws. Here, they find that byte models outperform their subword counterparts at fixed parameter counts, but not with matched training compute. Byte transformers demonstrate further improvement on a mixture of synthetic character manipulation tasks and common LLM and OCR benchmarks. Subsequently, the authors perform systematic ablations, finding that their byte transformers develop abstractions that loosely resemble an implicit tokenizer. Finally, the authors characterize the distribution of per-byte prediction difficulty and exploit the non-uniform structure for speculative decoding.

Critical strengths

Byte transformers outperforming subword transformers on the reported benchmarks is genuinely interesting. My initial intuition would have been that increasing the number of predictions to generate the same text might accumulate more error. I’m curious as to why this isn’t the case, and find these results useful for the community

Critical weaknesses

Because the underlying architecture and training recipes are different, one cannot wholeheartedly attribute the observed effects to the choice of bytes over tokens.

  • In particular, the reported byte transformers use longer GDN conv kernels, longer n-gram hash embeddings, and token-superposition training. While the authors ablate on these choices for training dynamics/scaling results, the mechanistic results are only evaluated over the full model.
  • How can we know that, for example, the formation of text abstractions actually occurs organically, and not because of the injected n-gram hash embeddings
  • The looped-subword experiments suggest that allocating additional compute towards subword models can recover the initial fixed parameter scaling gap. As such, why should one prefer byte-level language modeling beyond providing just another scaling axis? -- I know the point of this paper isn’t just to show capabilities, but I feel the authors should be forthcoming about this limitation, and further stress-test the result before including it as a main finding. More on this below

Depth > Breadth. The paper reads as if the authors pretrained a collection of models increasing in scale, performed several somewhat related experiments, and simply aggregated the findings into the paper. While admittedly, there does exist somewhat of a narrative around byte-level computation being useful as a whole, each subsection feels duly half-baked. Here are a few examples

  • Scaling -- See above points
  • Implicit tokenization -- The attention analysis is heuristic and performed on a small synthetic copying task. Further, instead of using representations identified from this analysis, the boundaries from frozen-layer experiments are determined by a separate model. At this point, it becomes unclear what the ensuing experiments are trying to accomplish, or more broadly, whether they actually validate the mechanism being described. The authors should consider tightening this analysis, making it more rigorous and systematic
  • Information allocation -- The spec decoding results are preliminary/proof of concept at best. First, they occur in open loop (oracle simulation), rather than actual decoding. The authors report 3.4x more accepted tokens, but neglect that the accepted number of bytes per target forward pass is almost identical to the subword counterpart. Again, the authors should be more forthcoming about both limitations here instead of sweeping them under the rug and hoping a reviewer doesn’t find them.

This paper, and the community I suppose, would greatly benefit from choosing one or two of the findings and understanding them with much greater depth.

Questions for authors

I’m reasonably concerned about whether or not the amount of source text is held fixed between the byte and subword regimes. Could the authors clarify whether their scaling comparisons are fair despite the TST exposure multiplier?

3 comments
AuthorsOpenReview ↗

Response to Reviewer badS (Part 1)

We thank the reviewer for the careful assessment and constructive suggestions, particularly regarding recipe attribution and the connection between our mechanistic experiments.

W1.1, Q1. Recipe differences and source-text exposure

Please refer to our General response for the recipe motivation and symmetric TST/hash comparisons. In Sec. 2.2, the IsoFLOPs sweep compares models under fixed training FLOPs, with the fits explicitly accounting for source-byte exposure, including TST’s multiplier.

For the main model suite, BYTE receives approximately 1.67× the source-byte exposure of raw SUBWORD. We will clarify Footnote 3: approximate exposure matching applies to ordinary next-token training, as stated in App. B.1.

Additionally, we note that the convolution widths and hash configurations provide comparable local text spans that account for the different input granularities: during ordinary next-token training and inference, BYTE’s conv size of 16 and SUBWORD’s conv size of 4 correspond to approximately 16 and 18 bytes, respectively. Since n-gram lengths are also measured in each family’s token units, using the same n would not match their text spans. BYTE uses n = 5–8 with four tables of 32,768 slots each, while SUBWORD uses n = 2 with one table of 131,072 slots. Thus, both families use the same total number of hash slots and matched text spans.

W1.2, W2.2. Evidence for useful local abstractions

To examine the role of TST and hash directly, we repeated the Figure 4 intervention on raw BYTE.

TaskMethod21-2321-2521-27
MMLUAttention84.978.774.3
MMLURandom76.564.558.9
CUTEAttention71.662.759.3
CUTERandom31.39.60.2

Although performance retention is lower than with TST+hash, the overall trend remains similar: compression in later layers preserves substantial performance and outperforms random segmentation at the same compression ratio. Importantly, this supports the emergence of useful local structures under ordinary BYTE training, while TST and hash further strengthen their effectiveness.

Regarding the connection between the experiments, the copying probe identifies attention patterns consistent with local aggregation, while the frozen intervention tests whether aggregated representations can support subsequent computation. Boundaries obtained from the smaller model remain useful for the larger target, suggesting transferability of local partitions across scales. Furthermore, retaining downstream performance with all parameters frozen provides functional evidence beyond the attention patterns themselves.

AuthorsOpenReview ↗

Response to Reviewer badS (Part 2)

W2.1. Central storyline and implications

Our central perspective is that byte-level modeling turns longer sequences into a useful scaling axis. Meanwhile, BLT and H-Net introduce local structure and improve efficiency through architectural changes, whereas we show that a flat Transformer can also learn useful local structures and offer opportunities for efficient inference.

In Sec. 2, we examine token granularity as a direction for allocating computation alongside model size and recurrent depth. The compute-optimal frontiers show that BYTE achieves lower BPB at the same optimal parameter count with more training compute, while SUBWORD remains better at the same compute budget. Together with the fine-grained downstream gains and the orthogonal scaling directions explored through Loop/MoE, these results establish the value and trade-offs of sequence-length scaling.

Beyond these performance gains, Sec. 3 examines whether the local structures explicitly introduced by BLT and H-Net can also emerge within a flat Transformer. The copying probe reveals repeated access to locally aggregated information, while the frozen layer intervention tests its computational usefulness: BYTE retains substantial downstream performance when selected layers operate on compressed representations. Together, these results demonstrate that a flat Transformer can also learn useful local structures that support subsequent computation. Furthermore, as shown in Sec. 3.2 and Figure 4, BYTE retains more performance when the compressed stage is placed later in the network. This suggests that more byte-resolution processing may be useful before compression, motivating further exploration of LGL layer allocation.

Nevertheless, fine-grained byte modeling still incurs higher inference costs. Sec. 4.1 therefore compares BPB distributions across granularities, showing that byte modeling distributes the same source entropy more unevenly, concentrating information on key bytes and offering finer control over computation allocation. This motivates the drafts in Sec. 4.2 to handle low-information continuations. The resulting comparable accepted text per simulated verification provides a proof of concept for speculative decoding of BYTE to reduce inference overhead.

Together, our findings establish sequence length as a useful scaling axis and show that a flat Transformer can also develop computationally useful local structures. Moreover, the drafting results offer a path toward reducing inference overhead, supporting a simple approach that retains the flat architecture while exploiting the benefits of byte-level modeling.

W2.3. Scope and limitations of the speculative-decoding experiment

We agree with the reviewer that this experiment is a proof of concept: the 3.4× result refers to accepted tokens on reference-text prefixes, and establishing practical speedups requires end-to-end evaluation on generated histories, accounting for drafting and verification costs. We already discuss these limitations in G.2 (Implications for speculative decoding), but space constraints led us to keep this discussion in the appendix. We recognize that these limitations should be stated more clearly and we have now stated these points explicitly in the Limitations section and clarified them in Sec. 4.2.

Our experiment uses the simplest setting with existing small models as drafts. Developing an optimized speculative-decoding system would require additional design and specialized draft training, which is beyond the scope of this paper. Our aim here is to test whether drafts can handle predictable byte continuations. As shown in Table 3, BYTE and SUBWORD achieve comparable accepted bytes per simulated target forward pass, suggesting that BYTE’s higher acceptance can compensate for its finer token granularity.

Thank you for your response. I will increase my score to 2.

Reviewer_BzYpOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work studies Byte models, comparing them to models using subwords (e.g. BPE tokens). First, a scaling law fitting process demonstrates the scaling potential of Byte and BPE models trained with and without TST and hash embeddings. The findings corroborate prior work in finding that BPE models are more compute efficient (achieve lower loss at the same compute, splitting compute between parameters and data optimally for that architecture), but Byte models may be able to take advantage of larger data budgets at fixed model size. Additional investigations of recursive (Loopformer) BPE and sparse (MoE) Byte architectures show that BPE Loopformer is between BPE and Byte models in compute efficiency and have optimal parameter count between BPE and Byte models, and that sparse Byte architectures do outperform dense Byte models, just as sparse BPE models outperform dense BPE models. Then, the work investigates the emergence of patching-like behavior learned implicitly by Byte-level models without explicit hierarchy. Byte level models trained on a copying task learn attention patterns which "focus" on a small set of specific byte positions, rather than only on the byte to be copied, suggesting that byte level models aggregate information in specific implicit "patch" boundary locations. When intermediate layers of the frozen Byte/BPE model are made to operate on "patch representations" (windows of representations corresponding to bytes, mean-pooled for the patch), with patch boundaries defined by an external entropy model, Byte models retain somewhat more performance compared to their BPE counterparts. Finally, there is an investigation of the model's internal allocation of information over bytes, specifically how a BPB model distributes the information of a single subword token over its multiple constituent bytes. Results indicate that Byte models may put more BPB mass on high and low BPB bytes, than BPE models put on high and low BPB BPE tokens, but that difference disappears if the bytes are aggregated into the same BPE tokens before computing the distribution, suggesting that Byte models distribute information similarly to BPE models over the same BPE divisions, but have greater flexibility within those BPE tokens.

Critical strengths

S1. Byte level modeling is of increasing interest in the field, so there's a growing need for analyses of this style.

Critical weaknesses

W1. The biggest difficulty I have with accepting this paper is the size, scope, and organization of the contribution. My understanding of the scaling experiments in S2 is that many of these results have been shown in prior work. The results of S3, while new to me, are somewhat lacking in implications for the area of Byte-level modeling. Similarly, S4 shows that Byte and BPE models do not significantly differ in allocation of information when bytes are grouped according to the BPE boundaries. The fact that Byte models have more flexibility to allocate between bytes is true by construction. Hence, I don't find that the sections as a whole successfully build up my understanding of Byte models, and I'm not convinced that the contribution is substantial enough to merit publication yet.

W2. Since the work goes to the effort of training multiple scales of models (Table 5), it's not clear why the authors don't choose to test whether the findings of S3-4 are consistent across the model scales.

Questions for authors

Q1. Re W1: What is the takeaway for research into Byte-level modeling? Is there any recommendation that you give on the basis of this work?

3 comments
AuthorsOpenReview ↗

Response to Reviewer BzYp (W1)

We thank the reviewer for the careful assessment and for recognizing the value of analyses of byte-level modeling. We appreciate the opportunity to clarify how the findings connect and what they suggest for future research.

W1, Q1. Central storyline and implications

Our central perspective is that byte-level modeling turns longer sequences into a useful scaling axis. Meanwhile, BLT and H-Net introduce local structure and improve efficiency through architectural changes, whereas we show that a flat Transformer can also learn useful local structures and offer opportunities for efficient inference.

In Sec. 2, we examine token granularity as a direction for allocating computation alongside model size and recurrent depth. The compute-optimal frontiers show that BYTE achieves lower BPB at the same optimal parameter count with more training compute, while SUBWORD remains better at the same compute budget. Together with the fine-grained downstream gains and the orthogonal scaling directions explored through Loop/MoE, these results establish the value and trade-offs of sequence-length scaling.

Beyond these performance gains, Sec. 3 examines whether the local structures explicitly introduced by BLT and H-Net can also emerge within a flat Transformer. The copying probe reveals repeated access to locally aggregated information, while the frozen layer intervention tests its computational usefulness: BYTE retains substantial downstream performance when selected layers operate on compressed representations. Together, these results demonstrate that a flat Transformer can also learn useful local structures that support subsequent computation. Furthermore, as shown in Sec. 3.2 and Figure 4, BYTE retains more performance when the compressed stage is placed later in the network. This suggests that more byte-resolution processing may be useful before compression, motivating further exploration of LGL layer allocation.

Nevertheless, fine-grained byte modeling still incurs higher inference costs. Sec. 4.1 therefore compares BPB distributions across granularities, showing that byte modeling distributes the same source entropy more unevenly, concentrating information on key bytes and offering finer control over computation allocation. This motivates the drafts in Sec. 4.2 to handle low-information continuations. The resulting comparable accepted text per simulated verification provides a proof of concept for speculative decoding of BYTE to reduce inference overhead.

Together, our findings establish sequence length as a useful scaling axis and show that a flat Transformer can also develop computationally useful local structures. Moreover, the drafting results offer a path toward reducing inference overhead, supporting a simple approach that retains the flat architecture while exploiting the benefits of byte-level modeling.

AuthorsOpenReview ↗

Response to Reviewer BzYp (W2)

W2. Consistency across model scales (For Sec 3)

We agree that extending the analysis across scales would further strengthen the evidence.

We first verified on the copying task that models across different parameter scales exhibit consistent attention aggregation. We then further conducted intermediate-layer compression experiments on the 700M and 3B models using two segmentation methods: segmentation based on the model’s own attention and random segmentation.

The results are shown in the table below, where the values denote the performance retention rate at the corresponding number of compressed layers (relative to the uncompressed model). As can be seen, our conclusion still holds for models of these two parameter scales.

TaskSizeMethod13-1513-1713-1913-2113-23
MMLU700M (24 layers)Attention77.675.274.173.773.1
MMLU700M (24 layers)Random62.659.258.257.958.0
TaskSizeMethod25-2925-3125-3325-3525-37
MMLU3B (40 layers)Attention91.389.687.487.387.0
MMLU3B (40 layers)Random88.484.677.980.777.7

W2. Consistency across model scales (For Sec 4)

For Sec. 4.1, we observe consistent BPB distribution patterns across our models. Furthermore, Figure 12 includes public models of different sizes and architectures, including BLT and H-Net. We believe this provides even stronger evidence of generality than validation restricted to our own model family.

For Sec. 4.2, we further extend the drafting experiment to 700M, 1B, and 3B targets with both 50M and 200M drafts. As shown below (also updated in paper G.3), BYTE and SUBWORD retain comparable accepted text coverage across target sizes, supporting the consistency of this finding.

Accepted bytes per simulated target forward pass (the higher the better). Drafts and targets belong to the same family.

Target size200M draft: BYTE200M draft: SUBWORD50M draft: BYTE50M draft: SUBWORD
700M (24 layers)10.21010.2605.6685.418
1B (28 layers)9.5279.5235.4715.111
3B (40 layers)8.3348.2015.1334.650

Dear Reviewer,

As the discussion period is coming to an end, we would like to kindly ask whether our responses and additional experiments have addressed your concerns satisfactorily. We would be happy to provide further clarification if needed.

Thank you again for your time and effort in reviewing our paper.

Best regards,

The Authors

Reviewer_Dv6COpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper asks whether the roughly 4x longer sequences of byte LMs can be used as extra computation instead of pure overhead, while keeping a flat Transformer (Qwen3.5-style hybrid attention, no BLT / H-Net style local-global-local hierarchy). Two training ingredients are co-designed with bytes: token-superposition training (TST, first 30% of training) and hashed byte n-gram embeddings (n = 5-8).

(1) Scaling: an IsoFLOPs sweep with Chinchilla-style fits, ablating raw / hash / TST / TST+hash for both families. At the same compute-optimal parameter count, byte models reach lower BPB than subword (Fig. 1a) and prefer roughly 1.5-3x more data per parameter (Fig. 1c). At the same training FLOPs, subword is still better (Fig. 1b).

(2) Downstream (200M-3B): big gains on fine-grained tasks (at 1B: CUTE 76.4 -> 99.6, OCRBench 31.5 -> 39.4, TextVQA 47.5 -> 52.4; Table 1) and small but consistent gains on 21 base-model QA tasks (Table 9). Looped subword and byte MoE are used to study how compute allocation shifts the gap (Sec. 2.4).

Critical strengths

Simple and elegant idea that finally makes byte LMs pay off.

Byte LMs are known to remove the tokenizer inductive bias, but the problem is we pay nearly 4x longer seq for nearly no gain in performance. This work thinks from another perspective: keep the flat Transformer and co-design the training recipe with bytes. TST is known to be compute-efficient but data-inefficient, which fits byte LMs well since we already got the longer seqlen (a 4-byte bag is roughly one subword token). And n-gram hash embeddings make a lot of sense when the vocab is only 256 and a single byte carries very little by itself. There is no new architecture, so it is easy for others to adopt. This is the main reason for my score: it turns the seqlen cost of byte LMs from pure overhead into extra compute per parameter.

Critical weaknesses
  • The main-suite comparison is not recipe-matched. All main-suite results (Table 1, Fig. 2, Tables 9/10/13/14) and Fig. 7 compare BYTE (TST+hash) with raw SUBWORD. Also, TST counts each 4-byte bag as one training token in the first 30% of updates (App. A.2), so BYTE sees more text. By my math from Table 7 and A.2, BYTE gets 1.9 x 1.05M ≈ 2.0M source bytes per update on average vs ≈ 1.2M for SUBWORD (262k tokens x 4.56 bytes), which is about 1.7x over training. Footnote 3 says the exposure is approximately the same, but App. B.1 says this only holds for ordinary next-token training.
Questions for authors
  • What is the actual total source-byte exposure of BYTE vs SUBWORD in the main suite once TST is counted? Footnote 3 and App. A.2 / B.1 read differently to me.
  • Do you have downstream results (Table 1, Table 9) for SUBWORD with TST+hash, even only at 1B? If BYTE still wins clearly, I'd be even more positive.
2 comments
AuthorsOpenReview ↗

Response to Reviewer Dv6C

We thank the reviewer for the thoughtful feedback and for highlighting our central perspective: longer byte sequences can provide useful extra computation per parameter. We address the exposure accounting and recipe-matched comparisons below.

W1, Q1. Source-byte exposure including TST

The reviewer’s calculation is correct. TST increases BYTE’s exposure by 1.9× relative to ordinary next-token training. With the batches in Table 7, the resulting BYTE/SUBWORD source-byte exposure ratio is approximately 1.67× over training.

We agree that Footnote 3 needs clarification and we have revised Footnote 3 to explicitly account for TST’s additional exposure.

Q2. Downstream results with TST+hash for SUBWORD

Please refer to our General response, where we provide QA RC results for both families under raw, hash, TST, and TST+hash using the IsoFLOPs models from Sec. 2.2. SUBWORD also benefits from TST+hash, while BYTE obtains larger gains overall. These results complement the BPB ablations and support the recipe motivation.

For the fine-grained tasks in Table 1, we further provide CUTE results below. Across the evaluated recipes, BYTE consistently achieves substantially higher scores than SUBWORD. This persistent gap supports token granularity as the main reason underlying the CUTE advantage.

SizeBYTE rawBYTE TST+hashSUBWORD rawSUBWORD TST
200M92.4199.1969.8967.17
400M99.8599.1870.7872.01
700M99.8399.5273.0474.85

Computational constraints prevent us from providing an additional SUBWORD TST+hash comparison and extending these comparisons to the multimodal tasks. Based on the General response, where TST provides most of SUBWORD’s recipe gains and adding hash has a relatively small effect, we expect SUBWORD TST+hash to perform similarly to SUBWORD TST.

Clarification on TST’s role

We appreciate the reviewer’s interpretation of how TST fits byte-level modeling and would like to clarify one point:

TST is known to be compute-efficient but data-inefficient, which fits byte LMs well since we already got the longer seqlen (a 4-byte bag is roughly one subword token).

We believe we share the view that TST can reduce computation for the same amount of training data. By "data-inefficient", we understand the reviewer to mean that TST processes more data under the same compute budget and it may not be the best way to describe TST.

Thank you for your response.

I think this is strong work that co-designs the byte LM in a clever way. Every paper naturally has some limitations, but I see this work offering new angles to advance tokenizer-free architectures field. In my humble opinion, I recommend clear acceptance.

Reviewer_7D74OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper researches byte-level autoregressive Transformers, hypothesizing that the additional sequence positions introduced by byte-level modeling provide useful computational capacity. It combines scaling experiments with downstream evaluations, mechanistic analyses of local abstractions, and speculative-decoding experiments. The empirical study is broad, revealing interesting findings, such as the strong performance on character-level and OCR tasks, and the intermediate-layer compression experiment. However, the paper does not establish a sufficiently novel architectural contribution, and the efficiency and performance claims seem to lack rigorous comparison against the existing literature on byte-level models.

Critical strengths
  1. The proposed BYTE model substantially outperforms the SUBWORD baseline proposed by the authors on CUTE and OCR tasks.
  2. The frozen layer intervention is interesting, demonstrating that later layers can operate on compressed local representations without retraining, with only relatively minor performance degradation.
  3. The distinction between parameter-matched and FLOP-matched comparisons is a valuable contribution, as this distinction matters for byte-level models dealing with substantially longer sequence lengths.
Critical weaknesses
  1. The architecture relies heavily on existing components like TST and hash embeddings. While demonstrating that local and global abstractions can emerge implicitly is an interesting angle, the paper does not sharply distinguish this finding from existing models that explicitly learn these structures (BLT, H-Net, ByteFlow). It also misses related work from the broader information-adaptive computation literature, such as CoBit. This makes novelty seem rather limited.
  2. The main downstream evaluations compare BYTE almost entirely against the authors' own SUBWORD baseline. Because the 1B BYTE model shares almost identical near-zero BPB exponents with BLT and H-Net, the information-allocation effect might just be a general property of byte models. The paper needs direct, matched comparisons against BLT, H-Net, and ByteFlow to prove actual SOTA performance.
  3. The efficiency claim seems to reverse depending on what you hold constant: BYTE wins on parameter count, but SUBWORD wins on equal FLOPs. To show that BYTE dominates existing approaches the paper needs Pareto fronts plotting performance against parameters, training/inference FLOPs, and wall-clock time.
  4. The default BYTE and SUBWORD comparisons are not comparable straightforwardly because they differ in TST, hash embeddings, receptive fields, and computation. A symmetric ablation would help isolate exactly how much of the performance gain comes from byte-level granularity versus these architectural additions.
  5. The 3.4× speedup is misleading because it reflects accepted tokens, not text throughput. In reality, the BYTE and SUBWORD models achieve almost identical accepted bytes/forward pass (9.527 vs. 9.523). Framing it as a true inference speedup without measuring drafting and verification costs is a stretch.
  6. Deriving segmentation boundaries from a separate 200M model and imposing them on the 1B model proves those local representations are useful, but it does not prove the 1B model discovers them independently. The study needs stronger controls, such as boundaries derived directly from the 1B model, random boundaries matched for span length, and baseline BPE boundaries to make the emergent tokenizer claim convincing.
Questions for authors
  1. Can you provide compute–performance Pareto fronts for BYTE, SUBWORD, BLT, H-Net, and ByteFlow, rather than relying on isolated matched checkpoints?
  2. How does BYTE compare against BLT, H-Net, and ByteFlow on the same downstream benchmarks when matched for parameters and FLOPs?
  3. How much of the BYTE advantage persists when TST, hash embeddings, receptive field size, and computation are controlled symmetrically across baselines?
  4. Can the 1B model successfully recover its own segmentation boundaries without relying on the separate 200M model?
  5. Does the 3.4× speculative-decoding result translate to tangible improvements when measured in accepted bytes, generated histories, wall-clock latency, and total draft-plus-verification cost?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 7D74

We thank the reviewer for the careful reading and constructive feedback, and for highlighting our results on fine-grained tasks, local abstractions, and parameter–compute trade-offs. We address the concerns below.

W1. Emergent local abstractions

The implicit structures identified in Sec. 3 differ from BLT and H-Net in how they arise. While these models introduce local–global organization through explicit architectural design, our flat BYTE model develops useful local abstractions during training. The frozen layer intervention further demonstrates that these representations can support compressed computation while retaining downstream performance.

Furthermore, our experiments reveal a difference in stage allocation. BLT favors a deeper local decoder, while H-Net uses a relatively symmetric local encoder–decoder design. In contrast, the effective global stage in our intervention lies closer to the end of the network (Sec. 3.2, Figure 4). This suggests that more local processing may be useful before global computation over aggregated representations, motivating further exploration of LGL layer allocation.

W2, Q1–Q2. Comparisons with existing byte models

We do not claim SOTA. Instead, we view the additional computation from longer sequences as a useful scaling axis worth investigating, as discussed in Secs. 2.2 and 4.

Regarding direct downstream comparisons, substantial differences in pretraining corpora and training amounts limit controlled comparisons of architectures. Given our limited computational budget, retraining these baselines under matched compute is beyond the scope of this paper.

ModelPretraining datasetTraining amount
BYTE-1B (ours)ClimbMix160B bytes
BLT-8BBLT-1T4.5T bytes
H-NetFineWeb-Edu-100B500B bytes
ByteFlow-1.3BFineWeb-Edu-100B500B bytes

W3, Q1. Parameter–compute trade-offs

We note that Sec. 2.2 and Figure 1 already report scaling behavior and fitted compute-optimal Pareto fronts over parameters and training FLOPs. BYTE achieves lower optimal BPB at the same optimal parameter count with more training compute, whereas SUBWORD achieves lower optimal BPB at the same FLOPs. This trade-off supports our perspective that longer sequences can provide useful additional computation at a given model size.

W4, Q3. Recipe design and convolution width

Please refer to our General response for why different token granularities motivate different recipes and for the symmetric TST/hash comparisons.

Additionally, we note that the convolution widths provide comparable local text spans: during ordinary next-token training and inference, BYTE’s conv size of 16 and SUBWORD’s conv size of 4 correspond to approximately 16 and 18 bytes, respectively.

W5, Q5. Speculative-decoding potential

Our experiment uses the simplest small-model draft setting as a proof of concept. As shown in Sec. 4.2 and Table 3, BYTE and SUBWORD achieve comparable accepted bytes per simulated target forward pass, suggesting comparable potential for joint verification despite their different token granularities.

We note that the 3.4× result refers to accepted tokens. Measuring end-to-end speedup, including drafting and verification costs, is beyond the scope of this experiment, and we acknowledge this limitation.

W6, Q4. Segmentation from a smaller model

We use a smaller model to define local partitions for two reasons. First, inspired by BLT’s use of a small model to guide patching, we use an architecturally matched 200M model and its token loss to identify boundaries (Sec. 3.2, Appendix E.2). Second, the target model’s loss may reflect more complex contextual information alongside local structure, so a smaller model may provide a simpler signal for local partitioning. The latter remains a hypothesis.

Furthermore, to address this concern directly, we examined whether the target model’s own attention yields effective partitions and compared them with random boundaries.

TaskMethod17-1917-2117-2317-2517-27
MMLU0.2B-bpb91.087.881.281.381.6
MMLUAttention95.193.390.890.590.3
MMLURandom89.577.569.567.268.4
CUTE0.2B-bpb87.374.264.756.658.2
CUTEAttention98.692.584.081.981.5
CUTERandom46.916.19.84.64.0

The values in the table denote the performance retention rate at the corresponding number of compressed layers (relative to the uncompressed model). It can be seen that the segmentation boundaries derived from the model’s own attention retain performance similar to those from the 0.2B model’s segmentation, whereas random segmentation leads to a substantial performance drop. This further demonstrates the existence of implicit structure within the BYTE model.

Thank you for the response!

The additional ablations and clarifications do help, especially the symmetric TST/hash experiments and the new attention-derived segmentation experiment. They address some of my earlier concerns, including the 3.4× result being an accepted-token diagnostic rather than an end-to-end speedup. I would encourage the authors to maintain this distinction in their claims.

However, there are still a few open questions on my side. While BYTE benefits more from TST and hash, the absolute QA performance is still below SUBWORD. Also, new experiments support the claim that useful local abstractions emerge internally, but do not fully isolate the correspondence to tokenizer-like linguistic units, nor do they prove advantageous over explicit segmentation approaches (BLT, H-Net, and ByteFlow).

The rebuttal helps clarify the scope, but I am still unsure about the novelty and comparative advantage. I will maintain my score.

AuthorsOpenReview ↗

Response to Reviewer 7D74

Thank you for the feedback!

We would like to further clarify that our central claim is that a flat Transformer can also support effective byte-level modeling through appropriate training adaptations, which was previously considered challenging due to the efficiency of long byte sequences. We thus establish a complementary route to improving byte LMs alongside explicit hierarchical architectures, rather than claim superiority over SUBWORD or existing hierarchical models.

At the same parameter count, TST+hash brings larger improvements to BYTE in both BPB and QA RC. In absolute terms, BYTE with TST+hash achieves better BPB, while SUBWORD retains higher QA RC scores in these comparisons. This gap may suggest that byte-level granularity is less well suited to these QA tasks. Nevertheless, BYTE demonstrates substantial advantages on CUTE and OCR-related multimodal tasks, highlighting the fine-grained capabilities that make this modeling direction valuable.

Our findings therefore encourage further work on training and computation allocation within flat byte Transformers, alongside continued advances in hierarchical architectures.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: Motivation behind different recipes

Several reviewers raised concerns about recipe differences between BYTE and SUBWORD and requested recipe-matched comparisons (7D74 W4, Q3; Dv6C Q2; badS W1, Q1). We clarify the motivation behind our recipe design and provide BPB and QA RC results for both families under raw, hash, TST, and TST+hash settings.

1. Different input granularities motivate different recipes

Addresses: 7D74 W4; Dv6C W1; badS W1

Tokenization provides SUBWORD with predefined local text units. Moving to BYTE shifts the organization of this local information into the model, motivating corresponding recipe adaptations. We use hash embeddings, wider convolution kernels, and TST to support local representation and aggregation, with TST also improving computational efficiency through token aggregation. Our different default recipes thus address the additional demands of organizing local information and processing longer sequences when learning directly from bytes.

2. Additional text exposure from TST does not guarantee improvement

Addresses: 7D74 Q3; Dv6C W1, Q1; badS W1, Q1

TST increases text exposure to approximately 1.9 times that of raw training. However, BYTE’s BPB even deteriorates with TST alone, whereas SUBWORD benefits from it. Further adding hash to TST yields improvements over raw for both families (fig1 and tables below). Thus, the benefits of TST’s additional text exposure are recipe-dependent and should not be taken for granted.

One possible explanation is that TST increases exposure while compressing inputs through embedding averaging, which obscures order within each bag. Hash introduces order-sensitive local features before averaging and may therefore help BYTE use the additional exposure more effectively. In contrast, SUBWORD tokens already encode local character combinations, which may explain why SUBWORD benefits from TST alone. The response to hash is also consistent with this interpretation: hash alone improves BYTE’s BPB, while its effect on SUBWORD is close to zero.

To make these comparisons more direct, we present key fitted results from Figure 1 in the table below (raw BPB and percentage changes relative to raw, the lower the better).

Model familyDepth / parametersrawhashTSTTST+hash
BYTEd16 / 191.9M1.102502−1.349%+0.763%−2.578%
BYTEd20 / 373.7M1.059760−1.280%+0.277%−2.518%
BYTEd24 / 644.4M1.026041−1.223%−0.118%−2.469%
SUBWORDd16 / 241.7M1.109793+0.032%−0.953%−1.347%
SUBWORDd20 / 435.8M1.067500+0.095%−0.951%−1.131%
SUBWORDd24 / 718.9M1.032864+0.149%−0.949%−0.947%

3. BYTE benefits more from recipe adaptation

Addresses: 7D74 Q3; Dv6C Q2; badS W1

As noted in Sec. 2.2, TST+hash yields larger BPB reductions relative to raw for BYTE than for SUBWORD across different sizes. We further provide QA RC results below (from the IsoFLOPs models in Sec. 2.2), where BYTE also achieves larger gains overall. These results extend the evidence to downstream performance and support adapting the recipe to BYTE’s modeling needs.

Model familyDepth / parametersrawhashTSTTST+hash
BYTEd16 / 191.9M40.34+0.62+0.42+2.36
BYTEd20 / 373.7M42.58+0.71+0.67+2.41
BYTEd24 / 644.4M44.41+0.78+0.88+2.45
SUBWORDd16 / 241.7M42.27+0.68+1.98+2.07
SUBWORDd20 / 435.8M44.77+0.38+1.92+1.70
SUBWORDd24 / 718.9M46.89+0.13+1.86+1.39

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.