- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduced a novel way to solve some known weaknesses (e.g., multi-digit arithmetic, character counting, etc.) of LLMs and attribute the cause to the unsuitable way of tokenization considering the nature tasks. To address the limitation, the authors propose a learnable decoder-only transformer as a retokenizer, which takes the canonical tokenization given an string as input and generates a better valid tokenization, aiming for a better downstream performance. Since all valid tokenizations are simply different combinations of existing tokens in the vocabulary that the model has seen during training, the authors claim that they solution is free of re-training or finetuning the LLM itself for adapting the learned retokenization. Empirically, this method is consistently better than the canonical tokenization, which suggests that a learned retokenization is superior than the general task-agnostic tokenization for LLMs on specific tasks. The authors also show that their learned retokenizations are generally shorter compared with other non-learned alternatives while achieving a better accuracy comparatively.
Critical strengths
- This paper revisits the idea of tokenization and proposes a learnable retokenizer to improve LLMs performance on certain tasks, such as multi-digit arithmetic and character counting.
- Compared with canonical and task-agnostic tokenization, this method shows a significant performance boost in Table 1, demonstrating its effectiveness on certain tasks that need a careful redesign of the tokenization.
- Retokenization does not require tuning the LLM itself. Since valid tokenizations are re-combinations of existing tokens in the vocabulary, the LLM can still comprehend the input even without extra finetuning.
Critical weaknesses
- Retokenizer is LLM- and task-dependent. Given a combination of a new LLM and a new task, retraining a retokenizer, according to my understanding based on the paper (Section 3.2), is necessary.
- It seems that tasks that are unsuitable for canonical tokenization, like multi-digit arithmetic and character counting, have been successfully solved by LLMs with thinking capability or with tool-using access (e.g., a Python interpreter).
- The retokenizer itself is a transformer, which may potentially introduce a huge overhead when the context increases and affect throughput and latency.
- Reusing a token in the vocabulary may still be at risk of introducing unrelated semantics into the context that may affect the LLM's understanding of the original task itself. In other words, string consistency does not always guarantee semantic consistency.
Questions for authors
- Is it possible to learn a transferable retokenizer that works for all tasks that can potentially benefit from retokenization?
- What are some advantages of the proposed method compared with LLMs with CoT reasoning capability and tool-using capability?
- What is the overhead and throughput with the retokenizer, and how does it perform with increasing length of input?
Thank you for your careful review and for highlighting the strengths of our approach. We address your questions and concerns below.
[W1,Q1] Transferring across tasks and models In the paper, we train a separate retokenizer for each model-task pair, but we now also investigate transfer across models and tasks. For models with different vocabularies, direct transfer is not possible because our method keeps the vocabulary fixed. Even when models share a vocabulary, transfer does not necessarily preserve the best performance. As shown in Fig. 12 in the updated App., transferring retokenizers between Llama 3B and 8B, in most cases, does improve the performance over canonical tokenization but does not match a retokenizer trained for the target model directly, as the effective tokenization depends on the internals and capabilities of the base model.
For transfer across tasks, we first note that some similarity between the training and target tasks is needed to expect generalization. As discussed in our responses to other reviewers, we investigate this through prompt perturbations and length generalization in arithmetic (Figs. 9 and 10). We also highlight that a separate retokenizer does not need to be trained for every task. In Fig. 13, we show that a single retokenizer jointly trained on multiple tasks achieves performance comparable to individually trained retokenizers on each task, although it requires more training iterations.
[W2,Q2] Comparison with reasoning and tool use We do not view our approach as a replacement for CoT reasoning / tool use or superior to them. Rather, we see retokenization as an orthogonal approach. Our goal is to provide a method for improving the use of any general pretrained model's existing capabilities, when they are hindered by tokenization, regardless of whether the base model supports CoT or tool use and regardless of our knowledge of the model internals. Moreover, leveraging CoT capabilities may require collecting or generating task-specific reasoning traces for training, which is not always readily available. Our approach instead uses task examples and generated answers to optimize tokenization directly, without requiring access to reasoning traces or changes to the base model.
[W3,Q3] Inference cost: We have added an inference cost analysis to the App. (Table 3), measuring retokenization latency and end-to-end throughput for Llama 8B on the arithmetic, acronym, and coding tasks.
As expected, the autoregressive retokenizer introduces additional latency compared to canonical tokenization, with the overhead increasing for longer inputs. However, its impact on end-to-end throughput varies across tasks.
For coding, where accuracy increases 25%, throughput decreases by 65% compared to canonical tokenization. However, we note that the learned retokenizer still achieves higher throughput than char-level tokenization, the hand-designed alternative with similar task performance but significantly lower tokenization latency.
For the shorter acronym inputs, the throughput reduction is more modest, at 15%. Interestingly, for arithmetic, throughput actually increases by 42%. This is because, for a small subset of expressions, canonical tokenization leads the model to generate unnecessarily long answers without reaching a solution, a behavior that we do not observe with learned retokenization.
Finally, coding is a particularly demanding case for measuring the overhead, since the LLM generates only a single token. For tasks requiring longer generations, we expect the cost of autoregressive generation with a multi-billion-parameter LLM to dominate the cost of the shallow neural retokenizer, reducing its relative impact on throughput.
[W4] Semantic consistency of alternative tokenizations We agree that preserving the input string does not guarantee that the model interprets its semantics in the same way under different tokenizations. However, there are two important considerations. First, our task-specific reward directly evaluates the effect of each tokenization on downstream performance. Tokenizations that introduce misleading associations or otherwise hurt the model's understanding of the task are therefore less likely to be selected during training. This also applies to canonical tokenization, which is not necessarily optimal or semantically consistent with the task objective. Second, our method only changes how the input string is segmented into existing vocabulary tokens. It does not introduce arbitrary tokens from the vocabulary unrelated to the input. Thus, while alternative segmentations can change how the model processes the input, they remain grounded in the original text.
Thank you again for your careful feedback. We hope our responses and additional experiments have addressed your concerns. We would be happy to discuss any remaining points further.