- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces a significant improvement of the H-Net Embedding-Tokenization approach using a two-level transformer that learns where to chunk/"re-tokenize" and embed a sequence. The idea is to adjust the tokenizer and embedding to the data and the model at once. Level 1 is a tokenizer+embedding for a byte encoded sequence. Level two is a classical transformer. Between level 1 and level two, there is a router that grades each token and only the top get selected, resulting in a fixed-rate compression. For the decoding part, the old tokens that got removed are reused and the once that got added are used in a weighted sum dependent on the router weight resembling a weighted skip connect around layer 2. For inference, a separate model is trained for the selection of tokens.
I especially liked section "4.5 TOKENS-AS-INPUT: A FLOP-MATCHED ρ SWEEP", where the authors show that they can compress SP-32k with either accelerated inference or improved performance. Would love to see more results on that.
Whereas the approach improved the state of the art in most cases and especially for larger models, it does not yet beat the SOTA for tokenize-embedders. However, it provides a variety of ablation studies to show strength and weaknesses and spark discussion.
This is a very interesting topic tackled by frontier labs and it would be good to have the code shared for reproducibility and to get a perspective on next step to beat the SOTA.
Critical strengths
- very compelling approach
- educational with a lot of ablation studies and details
- carefully designed experiments
- provides "tokenization" examples
Critical weaknesses
-
Reproducibility
- Code and logs/detailed results are not shared for reproducing and verification.
- Learning curves are missing.
-
no accounting for randomness
-
largest model size is 1B, reference sequence length is rather small with 512
-
somehow incremental to H-Net and no apple-to apple comparison against the original model but only their H-Net-EC variant
-
The paper claims that p=1 corresponds to a normal transformer. However, it is a transformer with a weighted skip connect around level 2.
Other issues:
- The optimization contribution is not clear and it does not connect to the intro.
- The paper starts that token chunking needs to be dynamic because different tokens have different relevance and information content and then enforces a fixed ratio compression.
- "Level 2 (f2). Operates only on the ⌈ρT⌉embeddings selected by the router, where ρ∈(0,1] is the target compression rate. May be any sequence-to-sequence model, including another MLT. Level 2 typically handles the bulk of the total computation."
- "The router produces a scalar score" It sounds like the router is only applied to a single token. However, token compression depends on the surrounding tokens. How is this incorporated?
- "We do not release code" I think this an issue for checking results as well as future applications. I encourage the authors to reconsider.
- some of the links seem broken and not directly linking to the subsection, where the claimed point is actually addressed.
Questions for authors
- What is the spread of randomness in the results?
- Can you share a learning curve comparison and accuracy curve comparison by training steps? Having no tokenizer would create likely lower performance at the beginning and then pick up later? I am curious if the picking up was finished?
- Is it possible to work with a warm up of MLT using a smaller model to safe flops and only afterwards switch to a big model? Assuming the hidden dim is the same, can MLT be transferred between models? I am thinking of training on a small model and then keep training while using it on a large model.
- How much effort goes into tokenization for SP?
- What would it take for this flexible tokenization concept to beat SP-32k when matching FLOPS?
- How does the difference change over training time? Does it grow or shrink?
- Why was the sequence length limited to 512 SP-32k tokens?
- How does performance change with sequence length?
- Whereas FLOPS matching is an interesting baseline, how does training time look like? Does it significantly degrade because of the larger sequence length for attention or the top-k routing?
- "4.5 TOKENS-AS-INPUT: A FLOP-MATCHED ρ SWEEP" looks super exciting and promising.
- Do you have more related data to share?
- Could this be the better approach for the future?
- In the paragraph starting with "Why compression can improve performance", you mention "attention dilution". However, level 1 uses attention on the uncompressed byte-level encoding.
- Aren't we wasting a lot of compute on something less working?
- Could some very local attention reduce memory and compute/Flops requirements? A window of 100 might suffice?
- How were parameters for the transformers decided on?
- Could they favor SP-32k?
- How does output of the model look like?
- Can you provide examples?
- Is there a danger that it removes tokens that are actually required for proper reading?
- How does the paper relate to "MRT5: DYNAMIC TOKEN MERGING FOR EFFICIENT BYTE-LEVEL LANGUAGE MODELS"?
- What are common tokenization approaches for DNA transformers? Please provide references.
- When reporting "p=1" results, is the Level 2 to level 1 weighting by the router applied?
- Where can I find more details on the causal router for inference?
- What are the speed measurements?
- How was the router threshold selected?
- How much does the inference setting align with the p parameter from training?
- How many seeds did you run?
4 comments
Thank you again for your detailed review and the many thoughtful questions. As the discussion period closes in a couple of days, we wanted to check whether our response answered them, and whether there is anything else you would like us to clarify. The revision now reports three-seed averages for the main results, with standard deviations in several tables, learning curves for the three-level comparison and the ablations, and more token-level results in Section 4.6, including converting a pretrained dense Transformer into an MLT. We would be glad to address any further comments in the time that remains.
weak accept
-
Thanks for the replies. I raised my score to weak accept. I won't go higher due to novelty, impact, and some open points also raised by other reviewers.
-
I am missing measurements on training speed and inference speed. For example Google's initial MoE paper worked with training steps but when I "inferred" the true training time that was needed, the results looked really bad. FLOPs counts hide the true timing behavior in the same way.
Thank you for raising your score and for the follow-up, which we really appreciate.
You are right that FLOPs do not show the full runtime cost, and part of the per-step slowdown in early MoE papers comes from routing at every MoE layer, where tokens are gathered and sent to experts on different chips via slow all-to-all transfers, whereas MLT only requires one gather and one scatter per level inside the same sequence and on the same chip -- there are no all-to-all transfers and routing is per level instead of per layer.
When we timed training steps on the same hardware, we found that at Medium, two-level MLT takes 0.66 s per step compared with 0.65 s for H-Net EC (about 2% slower) while three-level MLT is around 4% slower, down from 8% at Small, because most of this extra time comes from the router and residual predictor, whose compute share drops from 4% at Small to under 1% at XL.
Token-choice H-Net is about 17% slower at Medium because each sequence picks a different number of tokens. Efficient token-choice approaches typically compile one or more functions with different fixed array sizes to account for variable length sequences, typically up to 25% of total sequence length. This extra Level-2 compute and the ability to vary token counts per sequence help explain why token choice does better at matched FLOPs.
Thank you again for the thoughtful review and the helpful questions.
Thank you for the detailed review and many constructive questions. We are glad you liked the token-level results, which are now in Section 4.6 together with converting a pretrained dense Transformer into an MLT. Please see our general comment first, which summarizes the changes to the model.
Figure 4 tracks three-level against two-level MLT over training, and Figures 12 and 13 show learning curves for the ablations.
For scale and sequence length, we follow the scaling proxy of Wortsman et al. (2023), which fixes the reference length at 512 tokens (and the equivalent in bytes depending on the model size). Our compute budget limited us to 1.1B parameters, and we have not yet studied longer sequences.
Main results average three seeds, and several tables include standard deviations.
We added a token-choice H-Net with its ratio loss, which is closer to the original, and three-level MLT has lower BPB than it at Small and Medium (Table 16).
Thank you for catching the issue with ρ=1. The router starts with σ=ρ (=1), so an untrained model at ρ=1 is a standard Transformer, but σ is then trained, so a trained ρ=1 model can learn to mix in the skipped branch, which Section 3.1 now says. In this case the model keeps σ≈1 throughout training.
We rewrote the optimization contrbution (Section 1 and Appendix A). Each selected position computes both outcomes of its routing decision and the router learns from the difference, which gives a first-order approximation of the exact gradient without any sampling.
The number of selected positions is fixed per sequence, but which positions are selected, and how long each chunk is, depends on the content, with long chunks in predictable text and short ones in surprising text (Section 3.2). Section 5 discusses when a fixed rate is reasonable.
The router reads the encoder state, which already contains the preceding context through attention (Section 3.2).
Thank you for the note on links, and we went through the cross-references in the revision.
To answer your remaining questions: