- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper shows that token counts does not identify the token boundaries and that one can achieve perfect count prediction but fail at distinguishing between tokenizations.
Critical strengths
The construction showing that two tokenizations may always output the same number of tokens for any string yet have different tokenizations is nice and simple.
Critical weaknesses
- The flow of the paper is very inconsistent and reads more like notes rather than a paper with a story. For example, Section 5 introduces partial boundary information but never uses it anywhere else (or at least the connection is not clear from the main body of the paper). Figure 2, Table 1, 2 and 4 do not contribute much to the paper. The writing is very confusing. One could move these floats that do not do much for the paper to the appendix and instead be more careful in the writing, which sorely needs attention. The goal of the paper is to show the separation between counts and tokenizations, but this is not a surprising separation and apart from this separation not much follows from that.
- The main object of study of the paper is a very simple toy example: at most 8 tokens, 4 merge rules and an alphabet of 3. The paper mostly sweeps the exponential blow up of performing the experiments under the rug, omitting the time it would take to construct the possible candidates, which there are clearly exponentially many.
- It is unclear what the goal of the paper is. Although Theorem 2 is nice and simple, it is unsurprising that such tokenizations exist. My question to the authors is: What are the consequences (empirical or theoretical) of your contributions? Apart from the identifiability results, there is not much depth in what can be learned or used from this. The empirical results are shallow, over toy data, and do not address any practical issues in the field.
Questions for authors
What are the consequences, empirical or theoretical, of your contributions? What does this separation allow or disallow us to do?
What is the intuition behind Eq. 8?
Line 364: what does "least-resolved count class" mean? It states 80.51%, but this is not in the table.
Minor issues (no need to answer during rebuttal):
- Line 137: is used here undefined. I assume this is a special identifier for end of string?
- The writing is very confusing. For example, lines 145-147: "A finite transcript may leave several of them consistent [consistent with respect to what?] and the members of a single remaining class [remaining from what?] may still disagree on a task that does not identify [what is the definition of identify here?]." The paper uses a lot of qualifiers that add nothing to the text and in fact murks the meaning and goal of the sentence. This is indicative of either fully AI writing or heavy AI writing assistance. I highly encourage the authors to rewrite such sentences, as it distracts and disrupts the narrative flow and makes the paper much harder to follow.
- Other examples, Line 177: "Each ambiguous decision is local and has a short witness under segment access; count access discards precisely the information needed to resolve it." The last part of this sentence does not make sense to me. It is not that the count discards the information, but rather that it does not help with distinguishing. The sentence following that also adds absolutely nothing to the text, repeating the exact same statement as the previous one.
- Proposition 5 (line 209) is more a definition than a proposition. Is there a reason for this to be stated as a proposition?
- Line 255: cell here appears undefined. I suppose the authors mean the set of all tokenizers in that are consistent with the oracle's response for query (i.e. the numerator in Eq. 8). It would be better to be clear and unambiguous and state this to be what is meant by a cell.