acceptodds
Under review as a conference paper at ICLR 2027

B-Net: Learning Where to Compress in Byte-Level Language Models

Abstract

Hierarchical byte-level Transformers operate directly on bytes and group them into patches for efficient computation. Their patcher must identify useful boundaries without language-specific rules while keeping the resulting computational budget predictable. Space-based patchers encode language-specific structure, while learned patchers controlled by a soft ratio loss can deviate substantially from requested patch lengths. We introduce B-Net, a hierarchical byte-level Transformer that separates learned boundary placement from explicit compression control. A causal convolutional scorer learns input-dependent boundaries without a tokenizer or manually specified language features, while an adaptive threshold controller realizes the requested average patch rate. On English, the learned patcher recovers the quality of strong space-based segmentation without receiving spaces as boundary rules. The same patching mechanism remains effective in separately trained Chinese models. We compare B-Net with state-of-the-art byte-level Transformers under a FLOP- and data-matched protocol. B-Net outperforms the strongest byte-level baseline in each setting by 3.4 percentage points in mean downstream accuracy on English and 4.1 points in XWinograd accuracy on Chinese. It improves English WikiText performance from 0.85 to 0.81 bits per byte and matches the best Chinese validation bits per byte. These results show that learned byte patching can combine reliable rate control with superior predictive performance without manually specified language-boundary rules.

Primary area:
foundation or frontier models, including LLMs
Keywords:
byte-level language modeling, tokenizer-free language models, hierarchical Transformers, learned segmentation, adaptive computation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_exGKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes B-Net, a modification of H-Net that learns patch boundaries with a causal convolution and controls the average patch length through an adaptive threshold. The idea is to separate where to put boundaries from how many boundaries to keep. The experiments cover English and separately trained Chinese models, with results on compression stability, language modeling, and downstream tasks.

Critical strengths
  1. This paper identifies a practical discrepancy between requested and realized compression. Experiments and multiple compression targets provide strong evidence that B-Net reliably tracks the requested average patch length in the evaluated setting.

  2. The shared-backbone ablations are useful. In particular, B-Net gets almost the same English performance as the space-based variant without being given the boundary rule. This is a worthwhile result, even though it does not show that learned boundaries are better than the heuristic.

Critical weaknesses
  1. I am not fully convinced by the compute matching. The estimates use assumed patch lengths rather than the actual counts throughout training. More importantly, embedding lookups are counted as dense matrix multiplications, which adds much more estimated cost to BPE than to byte models. Since part of the paper’s argument is better quality at comparable compute, I would like to know whether the conclusion survives a more realistic accounting.

  2. The evidence for predictive improvements is less convincing than the evidence for rate control. The small English gain over BPE is only 0.2 points, and the Chinese downstream comparison mainly rests on 504 XWinograd examples. These gains may be real, but the reported point estimates do not tell us how stable they are. The existing multi-seed runs seem like a good opportunity to check this.

Questions for authors
  1. How do the comparisons change after correcting the embedding-lookup cost and using realized patch counts? Also, why do the configuration tables list H-Net compression as 6.0, while the seed study reports roughly 4.1–4.4?

  2. Could you report downstream scores and BPB across the existing seeds?

  3. Is the large-English mean computed over six tasks or seven? The text, Table 1, and Appendix G appear to use different versions of this aggregate.

8 comments
AuthorsOpenReview ↗

Response to Reviewer exGK - Part 1 of 2

We thank the reviewer for recognizing the importance of "a practical discrepancy between requested and realized compression" and finding "strong evidence that B-Net reliably tracks the requested average patch length". We also appreciate the reviewer's view that matching the space-based variant without the boundary rule is "a worthwhile result".

W1. FLOP matching

I am not fully convinced by the compute matching. The estimates use assumed patch lengths rather than the actual counts throughout training. More importantly, embedding lookups are counted as dense matrix multiplications, which adds much more estimated cost to BPE than to byte models. Since part of the paper’s argument is better quality at comparable compute, I would like to know whether the conclusion survives a more realistic accounting.

We follow H-Net’s [1] established analytical FLOP-counting protocol, including its input-embedding convention, to maintain comparability with prior work. The embedding-accounting convention affects the comparison with BPE, but does not explain B-Net’s gains over H-Net and the other byte-level baselines, which use the same byte vocabulary.

Our central contribution is improved patching over prior byte-level models. B-Net retains H-Net’s backbone while raising large-English mean accuracy from 51.7% to 56.1% and lowering WikiText loss from 0.85 to 0.81 BPB. Shared-backbone comparisons and component ablations establish the contribution of the patching mechanism.

W2. Consistency of improvements

The evidence for predictive improvements is less convincing than the evidence for rate control. The small English gain over BPE is only 0.2 points, and the Chinese downstream comparison mainly rests on 504 XWinograd examples. These gains may be real, but the reported point estimates do not tell us how stable they are. The existing multi-seed runs seem like a good opportunity to check this.

We have now evaluated downstream performance across five small-English runs. B-Net achieves 44.4% ± 0.6 percentage points mean accuracy, consistent with the reported 44.6%. Every run outperforms both SpaceByte (42.8%) [2] and H-Net (42.5%) [1]. The improvement over these byte-level baselines therefore persists across seeds.

These repeated results complement the large-English comparison, where B-Net reaches 56.1%, versus 51.7% for H-Net and 52.7% for SpaceByte. Our predictive-quality claim rests on consistent gains over prior byte-level models, rather than the small-English difference from BPE alone.

Q1. H-Net compression

How do the comparisons change after correcting the embedding-lookup cost and using realized patch counts? Also, why do the configuration tables list H-Net compression as 6.0, while the seed study reports roughly 4.1–4.4?

The configuration table and the seed study answer different questions. The configuration table supports a fair performance comparison, which requires each model to reach the compression that its FLOP budget assumes. The seed study tests whether a requested patch length reliably determines compression across random seeds (Section 4.2). With H-Net’s reported ratio-loss weight, the small seed-study models reach only 4.121 to 4.394 bytes per patch, averaged over five seeds. The H-Net baselines in our model comparisons therefore use a stronger ratio-loss weight (Section 4.2), which brings the large run to the target compression of 6.0 bytes on average (Appendix I, Table 16). SOMBRERO [4] observes the same shortfall with the reported weight and likewise raises it to keep the FLOP matching valid. This stronger weight does not hold H-Net back: when MrT5’s [3] PI controller sets the ratio-loss weight in the small H-Net, it reaches a similar 0.972 BPB, against 0.974 with a fixed weight (Appendix J, Table 19).

H-Net reaches the target only with a stronger ratio loss than reported, and under the reported setting every seed falls short.

Q2. Results across seeds

Could you report downstream scores and BPB across the existing seeds?

Yes. We have now evaluated five small-English B-Net runs on all seven downstream tasks and the full validation split.

B-Net runValidation BPB ↓Mean downstream accuracy ↑
Run 10.96443.4%
Run 20.94745.0%
Run 30.95044.3%
Run 40.94844.6%
Reported model0.94944.6%
Mean ± SD0.952 ± 0.00744.4% ± 0.6 pp

Every run outperforms SpaceByte (42.8%) [2] and H-Net (42.5%) [1] in mean downstream accuracy. The improvement over these byte-level baselines therefore persists across runs and is not confined to the reported checkpoint.

AuthorsOpenReview ↗

Response to Reviewer exGK - Part 2 of 2

Q3. Six or seven tasks

Is the large-English mean computed over six tasks or seven? The text, Table 1, and Appendix G appear to use different versions of this aggregate.

Thank you for spotting this inconsistency. The large-English mean averages seven tasks: LAMBADA, HellaSwag, PIQA, ARC-Easy, ARC-Challenge, XWinograd-en, and WinoGrande. We have corrected the descriptions so that the text, table caption, and appendix consistently identify the same seven-task evaluation.

References

  1. Hwang et al., “Dynamic Chunking for End-to-End Hierarchical Sequence Modeling”. In ICLR (2026).
  2. Slagle, “SpaceByte: Towards Deleting Tokenization from Large Language Modeling”. In NeurIPS (2024).
  3. Kallini et al., “MrT5: Dynamic Token Merging for Efficient Byte-level Language Models”. In ICLR (2025).
  4. Neitemeier et al., “SOMBRERO: Measuring and Steering Boundary Placement in End-to-End Hierarchical Sequence Models”. arXiv:2601.22805 (2026).

Thank you for the additional results.

However, my concern about the Chinese results remains unanswered. The additional English runs are helpful, but they do not show whether the Chinese gains hold across runs.

Reviewer_BPf9OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes B-Net, a hierarchical byte-level language model that keeps the H-Net backbone (Mamba-2 outer encoder/decoder, Transformer inner stack, EMA smoothing with copy upsampling) and replaces H-Net's patcher with two components: (i) a causal 1-D convolution with window 4 that maps each encoder state to a scalar boundary logit, and (ii) a single global threshold τ that is updated by an EMA (β = 0.999) toward the batch-mean of each sequence's n_target-th largest boundary probability, i.e., an online quantile tracker. Training begins with ~10% of steps using Top-K selection to calibrate τ, then switches to thresholding; a training-only capacity clip at 1.25× the target removes excess boundaries. At inference τ is frozen and selection is prefix-causal. The ratio loss of H-Net is dropped; the scorer receives gradient only through the EMA interpolation weights at selected positions.

Critical strengths
  • The rate-control experiments are carefully done and are the paper's most reliable contribution.
Critical weaknesses
  • The model dim and number of layers differ across baselines, so I don't think this should be called FLOPs-matched. For example, SpaceByte uses Transformer outer layers (T8+T16+T8, dim 768, 586M params), while B-Net uses H-Net's Mamba-2 outer layers (M5+T25+M5, dim 1024, 775M params). The outer architecture, width, depth and parameter count all change together, so you can't tell where the gain comes from. Even the FLOPs number is nominal: Table 11 lists H-Net at compression 6.0, but Sec 4.2 measures 4.1–4.4. The paper's own ablation (Table 8) shows the gain mostly comes from the backbone. If you keep B-Net's chunker and switch the outer layers to Transformer, the mean drops to 37.6–38.9, below SpaceByte (42.8). If you keep the backbone and use a simple space rule instead, you get 44.5 vs. 44.6 for B-Net. Fig. 4 also shows B-Net's boundaries match SpaceByte's exactly. So the only fair comparison is against H-Net, and there the chunker barely matters compared to a space-based chunker on the same backbone. The key proposal of the paper is not well justified. All downstream results are also single-seed, and the space-rule baseline is never run at the large scale.

  • Causal leak from Top-K warmup and capacity clipping. During warmup (~10% of training), boundaries are picked by Top-K over the full sequence. After warmup, capacity clipping also ranks over the full sequence. In both cases, whether position t is a boundary depends on future positions, which is the same leak as expert-choice routing. The model can learn to use this during training, but it isn't available at inference. The paper admits this in one sentence but never measures its impact, and honestly causal leak is not acceptable in LM pretraining research.

Questions for authors
  • Can you run the space-rule variant (B-Net backbone with a space-based chunker) at the large scale? For Chinese, can you also try a simple whitespace + UTF-8 character-start rule? These two experiments decide whether the learned chunker adds anything.
  • Can you compare against baselines with the same dim and number of layers, or at least the same backbone with different chunkers? Right now only H-Net is a fair comparison.
8 comments
AuthorsOpenReview ↗

Response to Reviewer BPf9 - Part 1 of 4

We thank the reviewer for describing our rate-control experiments as "carefully done" and "the paper's most reliable contribution". We agree that reliable rate control is central to B-Net, because it lets the FLOP estimate hold during training. Across five seeds, B-Net realizes 5.986 ± 0.015 bytes per patch on held-out data, against the six that its FLOP estimate assumes.

W1. FLOP matching across baselines

“The model dim and number of layers differ across baselines, so I don't think this should be called FLOPs-matched.”

FLOP matching controls computational cost, not architecture or parameter count. Different architectures generally require different widths and depths to reach comparable FLOPs per byte. We follow H-Net’s established analytical FLOP-counting protocol, with matched training data and byte budgets. Identical-backbone comparisons answer a complementary question: which patcher works better within the same architecture. We provide both complete-model and shared-backbone comparisons.

“The outer architecture, width, depth and parameter count all change together, so you can't tell where the gain comes from.”

B-Net and our large H-Net baseline have the same backbone, M5+T25+M5 with identical hidden dimensions. Replacing the patching mechanism raises mean downstream accuracy from 51.7% to 56.1% and lowers WikiText loss from 0.85 to 0.81 BPB. These gains come from replacing the patcher, not changing the backbone. Component ablations further separate the contributions of boundary scoring and compression control.

“Even the FLOPs number is nominal: Table 11 lists H-Net at compression 6.0, but Sec 4.2 measures 4.1–4.4.”

The configuration table and the seed study answer different questions. The configuration table supports a fair performance comparison, which requires each model to reach the compression that its FLOP budget assumes. The seed study tests whether a requested patch length reliably determines compression across random seeds (Section 4.2). With H-Net’s reported ratio-loss weight, the small seed-study models reach only 4.121 to 4.394 bytes per patch, averaged over five seeds. The H-Net baselines in our model comparisons therefore use a stronger ratio-loss weight (Section 4.2), which brings the large run to the target compression of 6.0 bytes on average (Appendix I, Table 16). SOMBRERO observes the same shortfall with the reported weight and also raises it to keep the FLOP matching valid. This stronger weight does not hold H-Net back: when MrT5’s PI controller sets the ratio-loss weight in the small H-Net, it reaches a similar 0.972 BPB, against 0.974 with a fixed weight (Appendix J, Table 19).

H-Net reaches the target only with a stronger ratio loss than reported. Under the reported setting every seed falls short.

“The space-rule baseline is never run at the large scale.”

We have now run this control. The space-rule variant achieves 56.8%, versus 56.1% in mean downstream accuracy for learned patching. B-Net recovers the English heuristic’s predictive quality without receiving its rules; we do not claim superiority over that heuristic on English. Learned patching also lowers validation BPB below space-like boundaries on three of the four additional languages and matches them on the fourth.

“All downstream results are also single-seed.”

Our large-scale reporting follows H-Net and AU-Net, which do not report across-seed variability for their main downstream comparisons. Each large-model run requires approximately one week on four RTX 6000 Blackwell GPUs. We now additionally report five small-model runs: 44.4% ± 0.6 percentage points mean downstream accuracy.

AuthorsOpenReview ↗

Response to Reviewer BPf9 - Part 2 of 4

W2. Causality of warmup and clipping

Causal leak from Top-K warmup and capacity clipping. During warmup (~10% of training), boundaries are picked by Top-K over the full sequence. After warmup, capacity clipping also ranks over the full sequence. In both cases, whether position t is a boundary depends on future positions, which is the same leak as expert-choice routing. The model can learn to use this during training, but it isn't available at inference. The paper admits this in one sentence but never measures its impact, and honestly causal leak is not acceptable in LM pretraining research.

We agree that this impact must be measured. We have now measured both mechanisms on small English and Chinese models (Appendix A.2 and A.3).

Capacity clipping is an optional memory safeguard: it prevents unusually high patch counts in individual sequences from exhausting GPU memory for the batch. This memory issue also affects other variable-length patching models, including H-Net. Our additional experiments show that clipping can be removed without degrading predictive quality in the tested settings.

Small English B-NetCapacity clippingRunsValidation BPB ↓
10% Top-K warmup (reported setting)Yes50.95–0.96
10% Top-K warmupNo30.95
5% Top-K warmupYes10.95
20% Top-K warmupYes10.95

Removing capacity clipping leaves validation BPB unchanged, and in Chinese it lowers BPB for both seeds tested (Appendix A.2, Table 4). Without clipping, Top-K warmup is the only noncausal training step. Halving or doubling its length has little effect on validation BPB: in Chinese, the 5%, 10%, and 20% models all reach 0.98 BPB (Appendix A.3, Table 5). The warmup itself remains noncausal, and the revised method section states that training is not strictly causal (Section 3.2).

All reported results use causal boundary selection. At inference, B-Net freezes the threshold and selects boundaries without ranking or clipping. Any reliance on future positions learned during training would therefore lower the reported numbers. Full-sequence Top-K during training also has a precedent: ByteFlow (§3.2) uses it throughout teacher-forced training, whereas B-Net limits it to the warmup.

B-Net’s reported quality does not depend on capacity clipping and changes little with the warmup length.

AuthorsOpenReview ↗

Response to Reviewer BPf9 - Part 3 of 4

Q1. Space-rule controls at large scale and on Chinese

Can you run the space-rule variant (B-Net backbone with a space-based chunker) at the large scale? For Chinese, can you also try a simple whitespace + UTF-8 character-start rule? These two experiments decide whether the learned chunker adds anything.

We have now run additional space-rule controls at large English scale and on Chinese, using SpaceByte’s byte-boundary rule within B-Net.

SettingPatching within B-NetValidation BPB ↓Downstream accuracy ↑
Large EnglishSpace-like rule0.7956.8%
Large EnglishLearned patcher0.7956.1%
ChineseSpace-like rule0.9756.7%
ChineseLearned patcher0.9858.7%

English accuracy is the seven-task mean; Chinese accuracy is XWinograd-zh. Comparisons use approximately matched analytical FLOP budgets and matched training data.

Learned patching recovers the English heuristic’s quality without receiving its rules. On Chinese, it achieves 2.0 percentage points higher XWinograd accuracy at nearly identical validation BPB.

Please compare this to the previous dynamic-patching models: B-Net outperforms every prior dynamic-patching model (H-Net, Dynamic Pooling with entropy or Gumbel boundaries, and ByteFlow) in our English and Chinese comparisons. Recovering the English heuristic’s quality with learned boundaries therefore represents an advance over the previous dynamic patchers.

We additionally trained separate models on 20B bytes per language for four further languages under approximately matched FLOP budgets:

Patching within B-NetVietnameseHindiArabicRussian
Space-like rule0.750.420.720.60
Learned patcher0.720.400.690.60
Learned patcher, no clipping0.720.420.690.56

Values are validation BPB; lower is better. Best displayed values are bold.

Learned patching lowers validation BPB below space-like patching on Vietnamese, Hindi, and Arabic, and matches it on Russian. Without clipping, B-Net also leads on Russian (0.56 vs 0.60). Together, these experiments establish its benefit: recovering effective English segmentation and adapting beyond it without manually specified language-boundary rules.

Q2. Shared-backbone comparisons

Can you compare against baselines with the same dim and number of layers, or at least the same backbone with different chunkers? Right now only H-Net is a fair comparison.

Our evaluation addresses two distinct questions. First, how does B-Net perform against state-of-the-art byte-level models? This requires comparing their complete architectures. Requiring identical backbones would change those models and prevent that comparison. We ensure a fair comparison through H-Net’s established analytical FLOP-matched protocol, with matched training data and byte budgets. FLOP matching controls computational cost, not architecture or parameter count.

Second, what does our patching mechanism contribute when the backbone is held fixed? We also provide this comparison. B-Net and our large H-Net baseline have identical backbone dimensions and layer counts (Appendix I, Table 16). Replacing the patching mechanism raises mean downstream accuracy from 51.7% to 56.1% and lowers WikiText loss from 0.85 to 0.81 BPB (Section 4.3, Table 1). Appendix G, Table 13 further compares boundary scorers, compression controllers, and space-like patching, supplemented by the new large-English and Chinese space-rule controls.

Complete-model comparisons show that B-Net outperforms prior state-of-the-art byte-level models; shared-backbone comparisons isolate the patching contribution. We provide both.

Reviewer_VKYCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors present B-Net, a hierarchical byte-level language model that is built on top of H-Net, and adds modifications of replacing its boundary detection and ratio loss with a convolutional router and adaptive threshold control. The paper demonstrates more stable compression rates and improved performance on English and Chinese benchmarks.

Critical strengths

The proposed method is simple and tested across different random seeds, achieving stable compression rates. The paper provides extensive baseline comparisons and interesting ablation studies. The method shows promising improvements in downstream performance.

Critical weaknesses
  • Limited novelty. The architecture is largely inherited from H-Net, and the technical contribution seems incremental.
  • Unclear benefit of using learned boundaries. The boundary ablation using spce-like delimeters achieves very similar performance compared to B-Net (44.5% vs. 44.6%), which makes it uncertain whether the learned patcher actually leads to meaningful advantages over simple delimiter based heuristics.
  • Baseline comparison concerns. From the visualizations in the paper, and the performance listed in the paper, H-Net seems to be performing awkwardly bad (e.g., drawing weird boundaries).
  • It is noted in the limitations section already, but the sophisticated tuning needed for training the model seems quite involved.
Questions for authors
  • What is the main difference between this paper and H-Net paper in terms of the experimental setup that's causing significant differences?
4 comments
AuthorsOpenReview ↗

Response to Reviewer VKYC - Part 1 of 2

We thank the reviewer for noting that B-Net is "simple", achieves "stable compression rates" across random seeds, and shows "promising improvements in downstream performance". We also appreciate the reviewer's positive view of our "extensive baseline comparisons and interesting ablation studies".

W1. Novelty

Limited novelty. The architecture is largely inherited from H-Net, and the technical contribution seems incremental.

We thank the reviewer for raising this point. Hierarchical byte-level models have two main components: the backbone and the patching mechanism. Our contribution is the patching mechanism. Improving the backbone is not the goal of this work. Existing backbones already perform well. Patching remained the missing piece: existing learned patchers did not consistently combine reliable compression control with competitive predictive quality, while successful alternatives such as SpaceByte [1] relied on manually specified boundary rules.

B-Net addresses this gap by separating boundary learning from explicit rate control. Keeping H-Net's backbone, our patcher raises large-English mean downstream accuracy from 51.7% to 56.1% and lowers WikiText bits per byte from 0.85 to 0.81 (Table 1), while reliably tracking requested patch lengths across the tested seeds and targets (Section 4.2). B-Net achieves the best results among the byte-level models in our English and Chinese FLOP- and data-matched comparison.

Retaining the backbone makes the benefit of replacing the patcher directly observable. The contribution resolves a practical limitation of learned patching while delivering substantial predictive gains without language-specific boundary rules.

W2. Benefit of learned boundaries

Unclear benefit of using learned boundaries. The boundary ablation using space-like delimiters achieves very similar performance compared to B-Net (44.5% vs. 44.6%), which makes it uncertain whether the learned patcher actually leads to meaningful advantages over simple delimiter based heuristics.

Matching the English space heuristic without receiving its rules is itself a contribution. In our large-English comparison, prior learned patchers (H-Net [2], Dynamic Pooling [3], and ByteFlow [4]) do not surpass SpaceByte [1], whereas B-Net does while retaining H-Net's backbone.

The benefit extends beyond English: our learned patcher adapts to other languages without manually redesigning boundary rules.

Patching mechanism within B-NetVietnameseHindiArabicRussian
SpaceByte's space-like rule0.750.420.720.60
Our learned patcher0.720.400.690.60
Our learned patcher, no clipping0.720.420.690.56

Values are validation BPB; lower is better. Best displayed values are bold.

B-Net learns dynamic boundaries without language-specific rules, matching space-like patching on English and Russian and surpassing it on Vietnamese, Hindi, and Arabic.

W3. H-Net baseline quality

Baseline comparison concerns. From the visualizations in the paper, and the performance listed in the paper, H-Net seems to be performing awkwardly bad (e.g., drawing weird boundaries).

Yes, we agree. The reviewer’s observation gets directly to the core issue: H-Net’s patching performs poorly. That is precisely the problem that B-Net fixes. Existing backbones already perform well. Our contribution is a patching mechanism with strong boundary placement and reliable compression control.

The H-Net authors themselves document this behavior. Their original paper [2] shows boundaries extending into the first characters of words in the first stage of the two-stage model (Figure 4). In a subsequent October 28, 2026 analysis, the team behind H-Net at Cartesia explicitly identifies shortcomings of the original model, termed “LH-Net,” including boundaries such as r|hino, mas|s, and Ove|r. They explain how this boundary-placement mechanism harms next-byte prediction and hierarchical modeling. The reviewer’s observation thus aligns with the H-Net team’s own subsequent diagnosis. Cartesia’s analysis

Our ablations confirm that both boundary scoring and compression control contribute to predictive quality: replacing the adaptive threshold with a ratio loss reduces mean accuracy from 44.6% to 42.7%; replacing convolutional scoring with cosine similarity reduces it to 43.0%. B-Net directly addresses the weakness of H-Net identified by the reviewer.

AuthorsOpenReview ↗

Response to Reviewer VKYC - Part 2 of 2

W4. Controller settings

It is noted in the limitations section already, but the sophisticated tuning needed for training the model seems quite involved.

Training B-Net required no sophisticated tuning. We used the same controller settings across small and large English models and Chinese models, and across target patch lengths of 4, 6, and 8. The target patch length specifies the desired compression, which fixed and learned patchers alike must set. Unlike H-Net, B-Net requires no auxiliary ratio-loss weight to balance against the language-modeling loss.

Our ablations support this: even freezing the threshold after warmup retains 44.0% mean accuracy, compared with 44.6% for the adaptive controller. Removing capacity clipping preserves or improves English and Chinese validation BPB. Clipping is a memory safeguard against out-of-memory errors, not a prerequisite for predictive quality. The default controller settings carry across model scales, languages, and compression targets without per-setting tuning.

Q1. Differences from H-Net’s setup

What is the main difference between this paper and H-Net paper in terms of the experimental setup that's causing significant differences?

We use H-Net’s FLOP-counting protocol. The key distinction is between final compression and the patch lengths actually realized during training. Final compression misses the additional computation consumed earlier when patches were shorter: a patch length of 4.5 rather than 6 sends one-third more patches through the inner network. Matching final compression therefore does not establish matched training computation.

B-Net realizes the patch length that its FLOP estimate assumes. Our analytical forward-FLOP estimates use the requested patch length of six (Section 4.1, Appendix K). Across five seeds, B-Net tracks six throughout training and reaches 5.986 ± 0.015 bytes per patch on held-out data. With H-Net’s reported ratio-loss weight, small H-Net models in the same study reach only 4.121 to 4.394 and process more inner positions than their estimate assumes. The H-Net baselines in our model comparisons therefore use a stronger ratio-loss weight, which keeps their compression close to the target (Section 4.2). Under this protocol, B-Net retains H-Net’s backbone and raises large-English mean accuracy from 51.7% to 56.1% by replacing its patching mechanism.

References

  1. Slagle, "SpaceByte: Towards Deleting Tokenization from Large Language Modeling". In NeurIPS (2024).
  2. Hwang et al., "Dynamic Chunking for End-to-End Hierarchical Sequence Modeling". In ICLR (2026).
  3. Nawrot et al., "Efficient Transformers with Dynamic Token Pooling". In ACL (2023).
  4. Deng et al., "ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer". In ICLR (2026).

With the discussion closing tomorrow, we would appreciate your assessment of our response. We added a multilingual comparison against the space-like rule and showed that the H-Net team's later analysis reports the boundary failures you observed. We also showed that one set of controller settings carries across scales, languages, and compression targets. We hope this evidence resolves your concerns about the benefit of learned boundaries, the baselines, and the tuning effort.

I thiank the authors for the response and the clarifications. However, I still have some concerns about the baselines. For example, the examples that the authors brought in the response is somewhat misleading; to my understanding, the Cartesia blogpost that the authors mentioned discusses boundaries in multi-stage H-Nets, whereas this paper evaluates single-stage models. So I'm not fully convinced this explains the differences in the reported results. I also still fidn the technical contribution somewhat incrementla, so I'll keep my original rating.

Reviewer_arE4OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces B-Net, a hierarchical byte-level Transformer model. Like other recent byte-level models such as BLT or H-Nets, B-Net splits bytes sequences into variable length patches which control how much computation is assigned to a given string. What makes B-Net different is how it learns to segment patches; the way boundary placement is learned and the way the desired compression rate is learned are separate. Two separate components are involved: (1) a convolutional boundary scorer that predicts a boundary probability at every byte position, which are used as patch weights during reconstruction in the model's next byte prediction loss; and (2) a compression rate controller that adjusts how many patch representations are created (and thus how much compute is assigned to the global model) by adjusting a patching threshold over training using EMA. Importantly, the compression rate controller simply finds the right threshold for the desired patch count/compute budget, and does not backpropagate through the model. The authors argue that these two components should be decoupled because boundary placement influences predictive quality, while patch count influences compute cost.

The authors also introduce a hierarchical model framework for experiments that allows easily swapping components of hierarchical models like H-Nets, MegaByte, SpaceByte, etc. in order to isolate the effects of each architectural component (such as patching) on language modeling and downstream task performance.

The authors find that B-Nets reliably targets a desired compression rate/compute budget when compared to H-Nets. It also outperforms all other tested byte-level model baselines and is competitive with BPE on English and Chinese language modeling/natural language tasks.

Critical strengths
  1. I generally found the paper to be quite easy to follow. The problem is well-motivated and the introduction is nicely written. The issue of controlling compression rates is an important unsolved problem in the area of hierarchical byte-level models.
  2. I quite liked the experimental setup in which the authors included both full-model comparisons to other architectures, as well as the experiments involving particular architectural component ablations.
  3. The results are quite impressive and show promise for byte-level models with learned boundary prediction and controllable compression.
Critical weaknesses
  1. This paper should engage with MrT5 (Kallini et al., 2025), a byte-level model that dynamically compresses byte sequences and used a controller to target a desired compression rate during training. The PI-controller used in that paper sounds very similar to the controller used by B-Nets, so MrT5 should be discussed at the very least, and perhaps the PI-controller formulation of Kallini et al. 2025 could be used as a baseline.
  2. In the discussion of compute efficiency, Compute Optimal Tokenization (Limisiewicz et al., 2026) should be included as it is the most relevant work on that topic. The related work should also engage with these other prominent byte level models: Hierarchical Autoregressive Transformers (Neitemeier et al. 2025); Bolmo (Minixhofer et al., 2025); Fast BLT (Kallini et al., 2026); MBP (Owodunni et al., 2026)
  3. The presentation of the paper would benefit from an architecture or methodology diagram in the main text.
  4. Arguably, the most useful application of adaptively setting thresholds and better boundary prediction in byte level models would be a multilingual setting, yet the experiments in the paper do not involve multiple languages. This limitation is noted by the authors, however, and would be appropriate for future work.
Questions for authors

I'm using this section for more minor comments/confusions regarding the text.

  1. The “Inference and causality” paragraph at line 202 was quite difficult to understand and should be rephrased.
3 comments
AuthorsOpenReview ↗

Response to Reviewer arE4 - Part 1 of 2

We thank the reviewer for finding the paper "quite easy to follow" and for recognizing compression control as "an important unsolved problem in the area of hierarchical byte-level models". We are glad the reviewer found our results "quite impressive" and valued the combination of full-model comparisons and component ablations.

W1. Comparison with MrT5

This paper should engage with MrT5 (Kallini et al., 2025), a byte-level model that dynamically compresses byte sequences and used a controller to target a desired compression rate during training. The PI-controller used in that paper sounds very similar to the controller used by B-Nets, so MrT5 should be discussed at the very least, and perhaps the PI-controller formulation of Kallini et al. 2025 could be used as a baseline.

We thank the reviewer for pointing us to MrT5 [1], which we have added to Related Work. Its PI controller adjusts an auxiliary deletion-loss weight; B-Net adjusts the boundary threshold directly, without a compression loss, separating rate control from boundary learning.

Because MrT5 is a bidirectional encoder-decoder continued from ByT5, we tested its gate and PI controller in B-Net using the same backbone, 20B English training bytes, and target patch length six (Appendix J, Table 19):

PatcherRate controlValidation BPB ↓Realized patch lengthMean accuracy ↑
B-Net scorerMrT5 PI controller1.125392.538.4%
MrT5 gateMrT5 PI controller1.120592.037.8%
MrT5 gateMrT5 PI controller + upsampling STE0.9685.942.7%
H-Net routingMrT5 PI controller + upsampling STE + ratio loss0.9725.942.6%
B-Net (ours)Adaptive threshold0.9496.044.6%

Without an upsampling straight-through estimator (STE), patches grow to several hundred bytes, because the PI controller clips its weight at zero and cannot lower the deletion rate once it exceeds the target.

With the STE, MrT5's gate reaches 5.9 bytes per patch, but obtains 0.968 BPB and 42.7% downstream accuracy, versus 0.949 BPB and 44.6% for B-Net. These comparisons support the benefit of B-Net's patching mechanism beyond reaching the target rate.

We also applied the PI controller to H-Net, where it sets the weight of H-Net's ratio loss. This H-Net stays on target at 5.9 bytes per patch and reaches 0.972 BPB and 42.6%, close to H-Net with a fixed ratio-loss weight (0.974 BPB, 42.5%). The PI controller thus keeps H-Net on target, but H-Net still trails B-Net by 0.023 BPB and 2.0 accuracy points.

W2. Additional related work

In the discussion of compute efficiency, Compute Optimal Tokenization (Limisiewicz et al., 2026) should be included as it is the most relevant work on that topic. The related work should also engage with these other prominent byte level models: Hierarchical Autoregressive Transformers (Neitemeier et al. 2025); Bolmo (Minixhofer et al., 2025); Fast BLT (Kallini et al., 2026); MBP (Owodunni et al., 2026)

We thank the reviewer for the suggested references. The revised version cites all five in Related Work (Section 2). Their contributions differ from our focus on learned patching with explicit compression control:

  • Compute Optimal Tokenization [2] uses BLT models to study how the compute-optimal number of bytes per patch changes with compute. We cite it in our discussion of computational efficiency.
  • Hierarchical Autoregressive Transformers [3] split text into words at Unicode whitespace and therefore belong alongside rule-based approaches such as SpaceByte.
  • Fast BLT [4] and MBP [5] speed up generation by predicting several bytes per decoding step and keep the patching of the underlying model.
  • Bolmo [6] converts pretrained subword models into byte-level models, whereas our comparison studies models trained from scratch.

Related Work states these distinctions. None of these works learns patch boundaries while controlling the compression rate. Our experiments compare patching mechanisms under a FLOP- and data-matched training protocol, and the suggested works broaden the context for that comparison.

W3. Architecture diagram

The presentation of the paper would benefit from an architecture or methodology diagram in the main text.

Thank you for this suggestion. We have added an overview diagram to the introduction (Figure 1, page 2), showing B-Net's architecture and the separation of boundary learning from compression control.

AuthorsOpenReview ↗

Response to Reviewer arE4 - Part 2 of 2

W4. Multilingual evaluation

Arguably, the most useful application of adaptively setting thresholds and better boundary prediction in byte level models would be a multilingual setting, yet the experiments in the paper do not involve multiple languages. This limitation is noted by the authors, however, and would be appropriate for future work.

We agree that multilingual modeling is a promising application. We have extended evaluation to four additional languages, training separate FLOP-matched models on 20B bytes per language.

ModelPatching / tokenizationVietnameseHindiArabicRussian
BPE TransformerSubword tokenizer0.790.750.740.60
SpaceByteSpace-like rules0.800.440.760.66
H-NetCosine scoring + ratio loss0.890.500.700.56
B-Net, space-likeSpaceByte's rule replacing our patcher0.750.420.720.60
B-Net (ours)Learned scoring + adaptive threshold0.720.400.690.60
B-Net (ours), no clippingOur patcher without the memory safeguard0.720.420.690.56

Values are validation BPB; lower is better. Best values at the displayed precision are bold.

B-Net's learned patcher achieves lower validation BPB than the space-like variant on Vietnamese, Hindi, and Arabic, and matches it on Russian, where B-Net without clipping is lower. It learns these boundaries without manually specified language rules. Joint training of a single multilingual model remains future work.

Q1. “Inference and causality” paragraph

The “Inference and causality” paragraph at line 202 was quite difficult to understand and should be rephrased.

Thank you for flagging this paragraph. We have rewritten it (Section 3.2, page 5) to clarify the distinction between training-time ranking and inference, where the frozen threshold ensures causal boundary selection.

References

  1. Kallini et al., "MrT5: Dynamic Token Merging for Efficient Byte-level Language Models". In ICLR (2025).
  2. Limisiewicz et al., "Compute Optimal Tokenization". arXiv:2605.01188 (2026).
  3. Neitemeier et al., "Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models". In ICLR (2025).
  4. Kallini et al., "Fast Byte Latent Transformer". In ICML (2026).
  5. Owodunni et al., "Dynamic Multi-Byte Prediction With Hierarchical Language Models". arXiv:2608.15454 (2026).
  6. Minixhofer et al., "Bolmo: Byteifying the Next Generation of Language Models". arXiv:2512.15586 (2025).

I thank the authors for the thorough additional experiments involving the MrT5-style gating and PI-controller, as well as additional multilingual evaluations. I will maintain my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.