- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes B-Net, a modification of H-Net that learns patch boundaries with a causal convolution and controls the average patch length through an adaptive threshold. The idea is to separate where to put boundaries from how many boundaries to keep. The experiments cover English and separately trained Chinese models, with results on compression stability, language modeling, and downstream tasks.
Critical strengths
-
This paper identifies a practical discrepancy between requested and realized compression. Experiments and multiple compression targets provide strong evidence that B-Net reliably tracks the requested average patch length in the evaluated setting.
-
The shared-backbone ablations are useful. In particular, B-Net gets almost the same English performance as the space-based variant without being given the boundary rule. This is a worthwhile result, even though it does not show that learned boundaries are better than the heuristic.
Critical weaknesses
-
I am not fully convinced by the compute matching. The estimates use assumed patch lengths rather than the actual counts throughout training. More importantly, embedding lookups are counted as dense matrix multiplications, which adds much more estimated cost to BPE than to byte models. Since part of the paper’s argument is better quality at comparable compute, I would like to know whether the conclusion survives a more realistic accounting.
-
The evidence for predictive improvements is less convincing than the evidence for rate control. The small English gain over BPE is only 0.2 points, and the Chinese downstream comparison mainly rests on 504 XWinograd examples. These gains may be real, but the reported point estimates do not tell us how stable they are. The existing multi-seed runs seem like a good opportunity to check this.
Questions for authors
-
How do the comparisons change after correcting the embedding-lookup cost and using realized patch counts? Also, why do the configuration tables list H-Net compression as 6.0, while the seed study reports roughly 4.1–4.4?
-
Could you report downstream scores and BPB across the existing seeds?
-
Is the large-English mean computed over six tasks or seven? The text, Table 1, and Appendix G appear to use different versions of this aggregate.
8 comments
Response to Reviewer exGK - Part 2 of 2
Q3. Six or seven tasks
Is the large-English mean computed over six tasks or seven? The text, Table 1, and Appendix G appear to use different versions of this aggregate.
Thank you for spotting this inconsistency. The large-English mean averages seven tasks: LAMBADA, HellaSwag, PIQA, ARC-Easy, ARC-Challenge, XWinograd-en, and WinoGrande. We have corrected the descriptions so that the text, table caption, and appendix consistently identify the same seven-task evaluation.
References
- Hwang et al., “Dynamic Chunking for End-to-End Hierarchical Sequence Modeling”. In ICLR (2026).
- Slagle, “SpaceByte: Towards Deleting Tokenization from Large Language Modeling”. In NeurIPS (2024).
- Kallini et al., “MrT5: Dynamic Token Merging for Efficient Byte-level Language Models”. In ICLR (2025).
- Neitemeier et al., “SOMBRERO: Measuring and Steering Boundary Placement in End-to-End Hierarchical Sequence Models”. arXiv:2601.22805 (2026).
Thank you for the additional results.
However, my concern about the Chinese results remains unanswered. The additional English runs are helpful, but they do not show whether the Chinese gains hold across runs.
Response to Reviewer exGK - Part 1 of 2
We thank the reviewer for recognizing the importance of "a practical discrepancy between requested and realized compression" and finding "strong evidence that B-Net reliably tracks the requested average patch length". We also appreciate the reviewer's view that matching the space-based variant without the boundary rule is "a worthwhile result".
W1. FLOP matching
We follow H-Net’s [1] established analytical FLOP-counting protocol, including its input-embedding convention, to maintain comparability with prior work. The embedding-accounting convention affects the comparison with BPE, but does not explain B-Net’s gains over H-Net and the other byte-level baselines, which use the same byte vocabulary.
Our central contribution is improved patching over prior byte-level models. B-Net retains H-Net’s backbone while raising large-English mean accuracy from 51.7% to 56.1% and lowering WikiText loss from 0.85 to 0.81 BPB. Shared-backbone comparisons and component ablations establish the contribution of the patching mechanism.
W2. Consistency of improvements
We have now evaluated downstream performance across five small-English runs. B-Net achieves 44.4% ± 0.6 percentage points mean accuracy, consistent with the reported 44.6%. Every run outperforms both SpaceByte (42.8%) [2] and H-Net (42.5%) [1]. The improvement over these byte-level baselines therefore persists across seeds.
These repeated results complement the large-English comparison, where B-Net reaches 56.1%, versus 51.7% for H-Net and 52.7% for SpaceByte. Our predictive-quality claim rests on consistent gains over prior byte-level models, rather than the small-English difference from BPE alone.
Q1. H-Net compression
The configuration table and the seed study answer different questions. The configuration table supports a fair performance comparison, which requires each model to reach the compression that its FLOP budget assumes. The seed study tests whether a requested patch length reliably determines compression across random seeds (Section 4.2). With H-Net’s reported ratio-loss weight, the small seed-study models reach only 4.121 to 4.394 bytes per patch, averaged over five seeds. The H-Net baselines in our model comparisons therefore use a stronger ratio-loss weight (Section 4.2), which brings the large run to the target compression of 6.0 bytes on average (Appendix I, Table 16). SOMBRERO [4] observes the same shortfall with the reported weight and likewise raises it to keep the FLOP matching valid. This stronger weight does not hold H-Net back: when MrT5’s [3] PI controller sets the ratio-loss weight in the small H-Net, it reaches a similar 0.972 BPB, against 0.974 with a fixed weight (Appendix J, Table 19).
H-Net reaches the target only with a stronger ratio loss than reported, and under the reported setting every seed falls short.
Q2. Results across seeds
Yes. We have now evaluated five small-English B-Net runs on all seven downstream tasks and the full validation split.
Every run outperforms SpaceByte (42.8%) [2] and H-Net (42.5%) [1] in mean downstream accuracy. The improvement over these byte-level baselines therefore persists across runs and is not confined to the reported checkpoint.