acceptodds
Under review as a conference paper at ICLR 2027

Multi-Level Transformer: A Minimal Transformer Extension for Learned Sequence Chunking

Abstract

Byte-level sequence models waste significant capacity on the inherent compressibility of raw data. Modern tokenizers address this by chunking byte sequences into tokens with roughly uniform information density, a heuristic central to scaling large language models. However, tokenizers are static, trained once, and blind to the downstream model. The Multi-Level Transformer (MLT) introduces a minimal modification to standard Transformers, augmenting them with hierarchical processing at different temporal resolutions controlled by a target compression rate. Unlike prior dynamic-chunking approaches that rely on ratio-balancing objectives or boundary heuristics, MLT enforces capacity by construction and recovers a standard dense Transformer at full rate. Applied to byte-level language modeling, MLT outperforms prior chunking mechanisms under matched compute and matches a tokenized Transformer trained on the same data, learning compression rates that reflect the differing information densities of English, Chinese, and DNA sequences. The discovered chunk boundaries closely match those of trained tokenizers on both English and Chinese, emerging purely from the language-modeling objective. Applied to token-level language modeling, MLT learns to skip 20–30% of tokens while maintaining baseline performance, yielding significant inference savings. Finally, MLT outperforms natural language tokenizers on DNA sequence modeling, demonstrating the flexibility of learned chunking beyond language domains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
dynamic chunking, hierarchical transformers, tokenizer-free language modeling, byte-level modeling, conditional computation, adaptive sequence modeling, efficient transformers, DNA sequence modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_o1nEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a significant improvement of the H-Net Embedding-Tokenization approach using a two-level transformer that learns where to chunk/"re-tokenize" and embed a sequence. The idea is to adjust the tokenizer and embedding to the data and the model at once. Level 1 is a tokenizer+embedding for a byte encoded sequence. Level two is a classical transformer. Between level 1 and level two, there is a router that grades each token and only the top get selected, resulting in a fixed-rate compression. For the decoding part, the old tokens that got removed are reused and the once that got added are used in a weighted sum dependent on the router weight resembling a weighted skip connect around layer 2. For inference, a separate model is trained for the selection of tokens.

I especially liked section "4.5 TOKENS-AS-INPUT: A FLOP-MATCHED ρ SWEEP", where the authors show that they can compress SP-32k with either accelerated inference or improved performance. Would love to see more results on that.

Whereas the approach improved the state of the art in most cases and especially for larger models, it does not yet beat the SOTA for tokenize-embedders. However, it provides a variety of ablation studies to show strength and weaknesses and spark discussion.

This is a very interesting topic tackled by frontier labs and it would be good to have the code shared for reproducibility and to get a perspective on next step to beat the SOTA.

Critical strengths
  • very compelling approach
  • educational with a lot of ablation studies and details
  • carefully designed experiments
  • provides "tokenization" examples
Critical weaknesses
  • Reproducibility

    • Code and logs/detailed results are not shared for reproducing and verification.
    • Learning curves are missing.
  • no accounting for randomness

  • largest model size is 1B, reference sequence length is rather small with 512

  • somehow incremental to H-Net and no apple-to apple comparison against the original model but only their H-Net-EC variant

  • The paper claims that p=1 corresponds to a normal transformer. However, it is a transformer with a weighted skip connect around level 2.

Other issues:

  • The optimization contribution is not clear and it does not connect to the intro.
  • The paper starts that token chunking needs to be dynamic because different tokens have different relevance and information content and then enforces a fixed ratio compression.
    • "Level 2 (f2). Operates only on the ⌈ρT⌉embeddings selected by the router, where ρ∈(0,1] is the target compression rate. May be any sequence-to-sequence model, including another MLT. Level 2 typically handles the bulk of the total computation."
  • "The router produces a scalar score" It sounds like the router is only applied to a single token. However, token compression depends on the surrounding tokens. How is this incorporated?
  • "We do not release code" I think this an issue for checking results as well as future applications. I encourage the authors to reconsider.
  • some of the links seem broken and not directly linking to the subsection, where the claimed point is actually addressed.
Questions for authors
  • What is the spread of randomness in the results?
  • Can you share a learning curve comparison and accuracy curve comparison by training steps? Having no tokenizer would create likely lower performance at the beginning and then pick up later? I am curious if the picking up was finished?
  • Is it possible to work with a warm up of MLT using a smaller model to safe flops and only afterwards switch to a big model? Assuming the hidden dim is the same, can MLT be transferred between models? I am thinking of training on a small model and then keep training while using it on a large model.
  • How much effort goes into tokenization for SP?
  • What would it take for this flexible tokenization concept to beat SP-32k when matching FLOPS?
  • How does the difference change over training time? Does it grow or shrink?
  • Why was the sequence length limited to 512 SP-32k tokens?
  • How does performance change with sequence length?
  • Whereas FLOPS matching is an interesting baseline, how does training time look like? Does it significantly degrade because of the larger sequence length for attention or the top-k routing?
  • "4.5 TOKENS-AS-INPUT: A FLOP-MATCHED ρ SWEEP" looks super exciting and promising.
    • Do you have more related data to share?
    • Could this be the better approach for the future?
  • In the paragraph starting with "Why compression can improve performance", you mention "attention dilution". However, level 1 uses attention on the uncompressed byte-level encoding.
    • Aren't we wasting a lot of compute on something less working?
    • Could some very local attention reduce memory and compute/Flops requirements? A window of 100 might suffice?
  • How were parameters for the transformers decided on?
    • Could they favor SP-32k?
  • How does output of the model look like?
    • Can you provide examples?
    • Is there a danger that it removes tokens that are actually required for proper reading?
  • How does the paper relate to "MRT5: DYNAMIC TOKEN MERGING FOR EFFICIENT BYTE-LEVEL LANGUAGE MODELS"?
  • What are common tokenization approaches for DNA transformers? Please provide references.
  • When reporting "p=1" results, is the Level 2 to level 1 weighting by the router applied?
  • Where can I find more details on the causal router for inference?
    • What are the speed measurements?
    • How was the router threshold selected?
    • How much does the inference setting align with the p parameter from training?
  • How many seeds did you run?
4 comments

Thank you for the detailed review and many constructive questions. We are glad you liked the token-level results, which are now in Section 4.6 together with converting a pretrained dense Transformer into an MLT. Please see our general comment first, which summarizes the changes to the model.

Figure 4 tracks three-level against two-level MLT over training, and Figures 12 and 13 show learning curves for the ablations.

For scale and sequence length, we follow the scaling proxy of Wortsman et al. (2023), which fixes the reference length at 512 tokens (and the equivalent in bytes depending on the model size). Our compute budget limited us to 1.1B parameters, and we have not yet studied longer sequences.

Main results average three seeds, and several tables include standard deviations.

We added a token-choice H-Net with its ratio loss, which is closer to the original, and three-level MLT has lower BPB than it at Small and Medium (Table 16).

Thank you for catching the issue with ρ=1\rho = 1. The router starts with σ=ρ (=1)\sigma = \rho \ (=1), so an untrained model at ρ=1\rho = 1 is a standard Transformer, but σ\sigma is then trained, so a trained ρ=1\rho = 1 model can learn to mix in the skipped branch, which Section 3.1 now says. In this case the model keeps σ≈1\sigma \approx 1 throughout training.

We rewrote the optimization contrbution (Section 1 and Appendix A). Each selected position computes both outcomes of its routing decision and the router learns from the difference, which gives a first-order approximation of the exact gradient without any sampling.

The number of selected positions is fixed per sequence, but which positions are selected, and how long each chunk is, depends on the content, with long chunks in predictable text and short ones in surprising text (Section 3.2). Section 5 discusses when a fixed rate is reasonable.

The router reads the encoder state, which already contains the preceding context through attention (Section 3.2).

Thank you for the note on links, and we went through the cross-references in the revision.

To answer your remaining questions:

  • We have not tried warm-starting from a smaller model, but converting a pretrained dense Transformer into an MLT (Section 4.6) shows that a router can be added to a trained model relatively cheaply.
  • Training SentencePiece is cheap, but it stays fixed afterwards and needs large embedding and output matrices over 32k tokens, which a byte-level model does not.
  • On bytes, the gap to SP-32k at matched FLOPs is mostly data, since SP-32k reads 1.5 to 1.8 times more bytes, and with the same data MLT matches it. On tokens, MLT already has lower held-out loss than the dense SP-32k model (Section 4.6). Across sizes the gap stays about constant (0.039 to 0.043 BPB, Table 3).
  • FLOP matching already includes the attention cost of the longer byte sequences (Appendix G).
  • The outer level has 2 to 4 of the 12 to 28 layers.
  • All models use the Wortsman et al. widths, which we did not tune for MLT.
  • No token is removed, since skipped tokens still go through Level 1 and receive Level-2 information through ff. The router mostly skips function words and punctuation, and Appendix M shows annotated bytes and tokens.
  • MrT5 deletes bytes inside a byte-level encoder, with a regularizer on the deletion rate, whereas MLT keeps every position, fixes the rate exactly, works as a causal decoder, and nests. It is now cited in Section 2.1.
  • For DNA, common tokenizations are k-mers (DNABERT), BPE (DNABERT-2), and single nucleotides (HyenaDNA, Caduceus), now cited in Section 4.5.
  • The causal router is trained to predict the top-kk decisions and thresholded at 0.5 (Section 3.4), and it agrees with top-kk on around 99.5% of positions and routes about the intended fraction.
  • The main results use three seeds.

Thank you again for your detailed review and the many thoughtful questions. As the discussion period closes in a couple of days, we wanted to check whether our response answered them, and whether there is anything else you would like us to clarify. The revision now reports three-seed averages for the main results, with standard deviations in several tables, learning curves for the three-level comparison and the ablations, and more token-level results in Section 4.6, including converting a pretrained dense Transformer into an MLT. We would be glad to address any further comments in the time that remains.

Reviewer_o1nEOpenReview ↗

weak accept

  • Thanks for the replies. I raised my score to weak accept. I won't go higher due to novelty, impact, and some open points also raised by other reviewers.

  • I am missing measurements on training speed and inference speed. For example Google's initial MoE paper worked with training steps but when I "inferred" the true training time that was needed, the results looked really bad. FLOPs counts hide the true timing behavior in the same way.

Thank you for raising your score and for the follow-up, which we really appreciate.

You are right that FLOPs do not show the full runtime cost, and part of the per-step slowdown in early MoE papers comes from routing at every MoE layer, where tokens are gathered and sent to experts on different chips via slow all-to-all transfers, whereas MLT only requires one gather and one scatter per level inside the same sequence and on the same chip -- there are no all-to-all transfers and routing is per level instead of per layer.

When we timed training steps on the same hardware, we found that at Medium, two-level MLT takes 0.66 s per step compared with 0.65 s for H-Net EC (about 2% slower) while three-level MLT is around 4% slower, down from 8% at Small, because most of this extra time comes from the router and residual predictor, whose compute share drops from 4% at Small to under 1% at XL.

Token-choice H-Net is about 17% slower at Medium because each sequence picks a different number of tokens. Efficient token-choice approaches typically compile one or more functions with different fixed array sizes to account for variable length sequences, typically up to 25% of total sequence length. This extra Level-2 compute and the ability to vary token counts per sequence help explain why token choice does better at matched FLOPs.

Thank you again for the thoughtful review and the helpful questions.

Reviewer_1d6JOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Multi-level Transformer (MLT), which aims to achieve top-k routing to selectively process byte-level sequences, and build a hierarchy. The authors show competitive results on English, Chinese, and DNA as well as potential computational savings on tokenized inputs.

Critical strengths

The architecture is simple and well-motivated. Also, the experiments cover multiple domains such as english, chinese and DNA, and multiple baselines and evaluation metrics are useful to estimate the effectiveness of the proposed method.

Critical weaknesses
  • Unfair baseline comparisons. The compression rate that is used for training all baselines is tuned for MLT.Another example of potential unfairness is -- H-Net is also adapted to MLT's top-k routing rather than evaluated in its original setting, making it difficult to assess whether MLT is indeed better than the baselines.
  • Causal routing concerns. The model uses non-causal top-k routing during training and a separate causal router at inference. The method that the authors propose does not fully address the mismatch between training and inference, or could be a presentation issue.
  • Evidence for efficiency claims is limited. The reported inference savings are theoretical, without actual latency or throughput measurements such as wall-clock time. MLT also still underperforms the tokenized baseline at matched training compute.
  • Missing multi-level cases. Despite the paper's motivation, recursively stacked MLTs are not evaluated end-to-end which makes readers get confused about if it is actually doable or not.

Overall, the submission is interesting, but the experimental evidences provided in the paper is not yet convincing enough to support its main claims.

Questions for authors
  • Can the authors compare MLT against independently tuned baselines, particularly H-Net in its original setting? An example experiment could be training MLT using H-Net's settings. This is probably a favorable scenario for H-Net as this paper does for MLT, but this would actually show how robust MLT could be in various training settings.
  • Can the authors provide actual inference-speed measurements and evaluate non-causal routing during training?
  • Have the authors tried stacking multiple MLT stages (say 2 stages or 3 stages) end-to-end? If so, what were the results? Is it feasible?
4 comments

Thank you for the review and the concrete suggestions. Please see our general comment first, which summarizes the changes to the model.

On fairness, we did not choose ρ=0.225\rho = 0.225 by tuning MLT, but from the compression rate of the tokenizer, since an SP-32k token covers about 4.5 bytes and 1/4.5≈0.2221/4.5 \approx 0.222 (Section 4.1). This rate turned out to work well for MLT (Table 1), and we have now added a ρ\rho sweep for H-Net EC showing that it is close to optimal there too (Table 9), so the shared rate does not favour MLT. To test H-Net closer to its original form, we added a token-choice H-Net trained with its ratio loss (Table 16), and three-level MLT has lower BPB than it at Small (1.061 vs 1.078) and Medium (1.015 vs 1.026), even though this H-Net routes 28 to 30% of bytes instead of 22.5% and so uses more compute. We kept our own implementation so that everything except the chunking stays fixed, which is what lets the leave-one-out ablation isolate each H-Net component (Appendix J).

You also suggested training MLT in H-Net's setting. The closest experiment we have is MLT with token-choice routing and H-Net's ratio loss in place of top-kk (Table 14), and with value weighting it matches standard MLT on two-level bytes, on tokens, and on three-level Medium bytes, so MLT does not depend on top-kk.

All reported numbers use the causal router, which agrees with top-kk on around 99.5% of positions. As in all expert-choice routing including Mixture-of-Depths, top-kk is non-causal during training (Section 3.4). Since the token-choice results above are causal during training as well, the mismatch between training and inference has little effect.

MLT adds only a small router, a gather and scatter, and the predictor ff to a standard Transformer. With the same data and compute it matches SP-32k in BPB (Table 3), where the data-matched reference now follows H-Net's procedure (Appendix B.3), while the standard SP-32k model reads more data at the same compute and stays ahead.

Thank you for asking about multi-level models. With the predictor ff, stacking works end to end, and three-level MLT has lower BPB than two-level MLT at every size from 85M to 1.1B (Tables 3 and 10) and in five of six languages (Table 2). Its first stage splits words into syllables and morphemes, and its second stag sends word starts to the deepest stage (Figure 1c). Please see the general comment for why ff makes this work.

Thank you again for your review. As the discussion period closes in a couple of days, we wanted to check whether our response and the revision addressed your concerns. In short, the revision adds a token-choice H-Net trained with its ratio loss (Table 16), MLT trained with token-choice routing and H-Net's ratio loss in place of top-k (Table 14), and end-to-end three-level models at every size and in six languages (Tables 2, 3 and 10). If there is anything else you would like us to clarify or test, please let us know and we will do our best to address it in the time that remains.

I appreciate the detailed response and additional experiments such as the nwe baseline comparisons and multi-level resutls. However, I still find the technical novelty limited, and also have the concern that the baselines might be being evaluated in an unfair setting. For exmpale, to my understanding, the H-Net baselines still use all-Transformer backbone as MLT, and have 2 layers for encoder and decoder, which seem to deviate from the original paper (correct me if I am wrong). Due to these concerns, I'll keep my original rating for now.

Thank you for the follow-up and for engaging with the revision.

To answer your question, our H-Net baseline does use two Transformer layers in the encoder and decoder where the original H-Net uses four Mamba-2 layers, but these are matched in size, since the H-Net paper itself counts one Transformer layer as matching two Mamba-2 layers of the same width. Every chunker in our main comparison runs in the same network at the same compute. We aimed for the closest sensible match to H-Net in a setting where only the chunking differs, which is also what the H-Net paper does when it reimplements SpaceByte inside its own architecture as SpaceByte++.

On novelty, we hope you might reconsider your assessment of the technical contribution. Your original review described the architecture as simple and well-motivated, and we do see these as the main contributions of the paper. H-Net relies on a cosine-similarity router, a smoothing module, a ratio loss, inter-stage normalization, a near-zero-initialized residual projection and Mamba-2 layers in its outer stages, and we spent much of this project finding out which of these are needed for scalable learned chunking. We tried several approaches with different levels of complexity and found that MLT needs none of them, and that the simplest and most principled approach worked best and holds up at every scale. MLT only adds a small router, a gather and a scatter, and a residual predictor to a standard Transformer. Because MLT is a standard Transformer in every other respect, the same architecture works on bytes and on tokens, nests into multiple levels, and can be obtained from a pretrained dense Transformer by finetuning. We also put a lot of effort into understanding why it works, and the appendices analyse which parts of MLT and of H-Net help the router learn boundaries and which prevent it from doing so. We hope this analysis will be as useful to people building on H-Net as to those building on MLT.

We would also gently note that the concern about novelty does not point to any prior work that already does this, which makes it hard for us to respond to concretely. If there is a method you have in mind, we would be very grateful for the reference and will gladly discuss it in the final version.

Thank you again for your time.

Reviewer_LNaBOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MLT, a two-level Transformer for learned chunking. A shallow encoder runs over all positions, an MLP router scores each position, and the top ⌈ρT⌉\lceil \rho T \rceil positions go through a deeper Level-2 Transformer. The outputs are scattered back and mixed with the encoder residual. At inference, a separate causal router replaces top-k. The authors train 85M–1.1B models on bytes (C4 en/zh, HG38) and on SP-32k tokens, FLOP-matched to a dense SP-32k model. They compare against Random, Space, ByteFlow and an H-Net re-implementation, and also measure how well the learned boundaries match SP-32k token boundaries.

Critical strengths

The setup is simple and well controlled. Setting ρ=1 gives back a dense transformer, there is no auxiliary ratio loss, and the H-Net baseline uses the same skeleton and training recipe. Because of this, the leave-one-out ablation actually isolates H-Net's chunking components. It shows that most of that extra machinery is not needed, which I think is the most useful takeaway of the paper.

Critical weaknesses
  • Limited novelty. Learning chunk boundaries end-to-end with a neural scorer has a long history. Examples include history compression (Schmidhuber, 1992), HM-RNN (Chung et al., 2017), which learns binary boundaries with STE and updates the upper level only at boundaries, Skip RNN (Campos et al., 2018), Dynamic Token Pooling (Nawrot et al., 2023) and MrT5 (Kallini et al., 2024). Compared with this line of work, MLT mostly replaces a threshold or ratio loss with top-k. HM-RNN, Skip RNN and MrT5 are not cited at all. Also top-k as we know leak future info which is not acceptable.
  • The results rely on a non-causal training setup and on sweeping ρ, and they are still not strong. Top-k over the full sequence makes each routing decision depend on future positions, so training leaks future information. The causal router only approximates this after the fact. Also, ρ is tuned separately for each domain (0.225 / 0.3 / 1/6), and the baselines just reuse MLT's best ρ. So "learning the compression rate" is really a hyperparameter search. Even with this tuning, MLT only matches SP-32k in the data-matched setting, where the SP-32k run is cut to 56k–68k steps. At matched total compute it is clearly behind (XL: BPB 1.006 vs 0.952, LAMBADA 40.7 vs 43.9). Gains over H-Net EC are at most about 2 points, with no variance reported.
Questions for authors

Are the BPB numbers in Tables 1–4 computed with non-causal top-k or with the causal router? If top-k, please report causal-router BPB.

3 comments

Thank you for the careful review and for pointing out several gaps in the submitted version, which we hope the revision now addresses. Please see our general comment first, which summarizes the changes to the model.

On novelty, we agree that learning boundaries end to end is an old idea, and we do not claim it, since H-Net, the closest prior work, already learns boundaries end to end at scale. Our contribution is a simpler and improved way to train the boundary decisions. MLT needs no ratio loss, similarity-based boundary detector, smoothing, or inter-stage normalization, and its outer residual stream stays that of a standard Transformer. In a controlled comparison with the same network and training setup, MLT has lower BPB than H-Net at every size from 85M to 1.1B (Table 3), and adding H-Net's cosine router or its inter-stage normalization to MLT stops the router from finding token boundaries and raises BPB by about 0.025 (Table 17). Thank you also for the references, and Section 2.1 now cites HM-RNN, Skip RNN, and MrT5, next to history compression and Dynamic Token Pooling.

MLT also trains the boundary decision differently from the works you cite. HM-RNN and Skip RNN use a straight-through estimator, Dynamic Token Pooling uses a Gumbel-sigmoid with a prior on the boundary rate, and MrT5 uses a soft deletion gate with a rate regularizer. In MLT, each selected position computes both outcomes of its decision, the Level-2 output and what it would have received if skipped, and the router learns from the difference (Appendix A). The ablations show that this comparison has to be exact, since if the "skipped" outcome does not match what skipped positions actually receive, BPB rises by 0.034 and downstream scores drop (Table 17), and a straight-through gate is also worse (Table 18).

Because the residual stream stays standard, a pretrained dense Transformer can be converted into an MLT (Section 4.6), whereas with H-Net's merge the converted model ends 0.08 to 0.14 nats worse (Table 13). On top of a tokenizer, MLT also skips 10% to 40% of SP-32k tokens in the deep layers while lowering held-out loss.

All reported numbers use a causal router, which agrees with top-kk on around 99.5% of positions. You are right that top-kk is non-causal during training, as in all expert-choice routing including Mixture-of-Depths, and Section 3.4 now discusses this. We also trained with token-choice routing, where each decision depends only on the past, and with value weighting alone it matches top-kk. The gains therefore do not come from seeing the future.

We did not choose ρ=0.225\rho = 0.225 by tuning MLT, but from the compression rate of the tokenizer, since an SP-32k token covers about 4.5 bytes of C4/en and 1/4.5≈0.2221/4.5 \approx 0.222 (Section 4.1). This rate turned out to work well for MLT (Table 1), and we have now added a ρ\rho sweep for H-Net EC showing that it is close to optimal there too (Table 9), so the shared rate does not favour MLT. We agree that ρ\rho is a hyperparameter, and the revised abstract no longer says that MLT learns it, since MLT learns where to place boundaries while ρ\rho sets how many.

Thank you for pointing out the issue with the SP-32k comparison. The data-matched reference in the submitted version came from a misunderstanding of how H-Net makes this comparison fair, and we now follow the same procedure, where the reference reads the same bytes per sequence as MLT and we add layers until its compute matches again (Appendix B.3). With the same data and compute, three-level MLT matches or beats it in BPB at every size (XL: 0.936 vs 0.939), while the standard SP-32k model, which reads 1.5 to 1.8 times more bytes at the same compute, stays ahead (XL: 0.893), as Section 4.3 says.

Finally, main results average three seeds and all BPB numbers in Tables 1 to 5 use the causal router.

Thank you for the thorough response and revision.

The causal-router clarification, the token-choice experiments, the revised data-matched comparison, and the more careful framing of ρ address most of my soundness concerns, and the controlled ablations against H-Net’s components are a useful contribution to the dynamic-chunking literature. I am therefore raising my score to weak accept.

I still have some reservations about novelty: the routing mechanism is essentially expert-choice top-k as in Mixture-of-Depths, placed inside an H-Net-style encoder–decoder, and the main new component (the residual predictor f) has a small effect in the two-level setting (Table 17). But generally I think the paper provide some interesting insights to study,

Thank you for reading the revision carefully and for raising your score, we really appreciate it.

On novelty, the router is indeed expert-choice top-k as in Mixture-of-Depths, but MoD decides separately in every layer whether a token skips that layer, whereas MLT makes a single decision before a deep block of layers. The selected positions therefore form one compressed sequence that all Level-2 layers share, which is what turns the routing decisions into chunk boundaries and lets the levels nest. Compared with H-Net, MLT drops the ratio loss, the cosine-similarity router, the smoothing module, the inter-stage normalization and the near-zero-initialized residual projection, and still has lower BPB in a controlled comparison (Table 3).

We also find two results especially exciting. Trained only to predict the next byte, MLT places its boundaries at SP-32k token starts with a strong alignment without ever seeing a tokenizer (Table 4). And because its residual stream stays that of a standard Transformer, the same architecture also runs on top of a tokenizer, and a pretrained dense Transformer can be converted into an MLT in place by finetuning. This is where f matters, since its effect in the two-level byte setting is small (Table 17), but without it three-level models fall behind two-level ones (Section 4.2).

As the discussion period closes soon, please let us know if there is anything else we could clarify or add in the next couple of days.

Comments on the paper

1 comment

Thank you all for the careful and constructive reviews. They led to a substantial revision. Some of the changes affect the model itself, so we summarize them here once, and each individual response refers back to this comment.

The main change to the model is a small predictor for skipped positions (Section 3.2, Eq. 2). A selected position still receives the Level-2 output. The other branch, which is what a position receives when Level 2 skips it, changes from the encoder state EtE_t to Et+f(Et,M)E_t + f(E_t, M). Here MM is the latest Level-2 output and ff is a one-layer MLP.

This change unlocks three improvements.

  1. It allows training deeper hierarchies which we analyze in detail in Section 4.2 and Appendix D. Three-level MLT has lower BPB than two-level MLT at every size from 85M to 1.1B (Tables 3 and 10) and in five of six languages (Table 2).
  2. A pretrained dense Transformer can be converted into an MLT (new Section 4.6). ff only adds a correction, so the residual stream stays that of the original Transformer. With 20k finetuning steps, the converted MLT comes within 0.02 to 0.04 nats of an MLT trained from scratch.
  3. Standard training no longer needs value weighting. The submitted version always weighted Level-2 attention values by the router score, which gave the router extra learning signal. With ff, the comparison between the two branches already gives enough signal, so value weighting is now off by default. We keep it where the signal is otherwise weak, namely token-choice routing and Dense →\rightarrow MLT finetuning.

For stability, we also found that adding small random noise to the router scores during the first 1,500 training steps helps. Without it, some runs settle early into a worse routing pattern, especially at deeper levels (Appendix K, Table 18).

Beyond the model, main results now average three seeds, and the data-matched SP-32k reference now follows H-Net's procedure (see our response to Reviewer LNaB). We take ρ\rho from the tokenizer (Section 4.1). We added token-choice routing comparison, which is fully causal during training, a token-choice H-Net with its ratio loss, six mC4 languages, and the references the reviewers suggested.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.