acceptodds
Under review as a conference paper at ICLR 2027

How Far Should Tokenization Go? Predictive Effectiveness and Relational Losslessness

Abstract

GPT-style models have achieved remarkable success with finite vocabularies of reusable tokens, making the token interface a central component of modern sequence modeling. Symbolic music appears naturally compatible with this paradigm: it consists of discrete note events and recurring structures such as chords, motifs, and phrases. However, when tokenization moves beyond language, the interface must be specified for each domain. Existing work offers many effective designs, but no unified criterion for deciding what tokenization should represent and how far it should go. Using predictive codelength as a common criterion, we formulate the Effectiveness–Losslessness Framework to define where tokenization should begin and where it should end. The Fact–Token Boundary marks where observation-determined structure should enter the token interface, through operations such as coordinate construction. Within this interface, the resulting carrier may be reversibly recoded, for example through BPE, without changing the represented facts. The Token–State Boundary marks where tokenization should stop: relations that depend on context should remain for model-state computation rather than being fixed in advance by the tokenizer. We validate the framework through controlled multi-seed symbolic-music experiments, with an independent-corpus replication of the temporal intervention. Making musical time explicit consistently reduces predictive code and also improves pitch and duration prediction, while tonal-frame canonicalization and pitch factorization provide further gains. Fixed circle-of-fifths pitch coordinates instead increase predictive code, suggesting that imposing a fixed pitch relation before context can burden prediction. Reversible BPE substantially shortens the carrier but increases predictive codelength in every seed, showing that carrier compaction alone does not guarantee predictive gain. Together, these results show that tokenization quality is determined neither by recoverability nor by sequence length, but by where predictive computation is placed. The framework therefore recasts tokenization as a model-relative boundary: static representation exposes reusable regularities, while contextual state resolves what remains context-dependent.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
tokenization, representation learning, predictive coding, symbolic music, autoregressive modeling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RjPLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an Effectiveness–Losslessness Framework for allocating structure between token representations and contextual computation. It distinguishes coordinate construction, reversible carrier coding, and fixed relational geometry, comparing them through predictive codelength per declared musical fact. Symbolic-music experiments report benefits from temporal coordinates and canonicalization, a penalty from the tested circle-of-fifths interface, and a case where reversible BPE shortens sequences but increases predictive codelength.

Critical strengths
  1. The division of work between representation and contextual modeling is a worthwhile research question. Separating information preservation, sequence compaction, and predictive usefulness provides a useful way to organize comparisons.

  2. Symbolic music is a suitable controlled testbed. The common fact-level accounting includes termination and inverse-shift costs, and the temporal-coordinate finding receives supporting replication on an independent corpus.

Critical weaknesses
  1. The framework's additional value as a design principle needs clearer support. Sections 2.2–2.5 offer a useful classification of operations, but their acceptance ultimately depends on whether measured codelength improves. I remain unsure what guidance the proposed boundary adds beyond comparing representations under a specified model and budget. The model-relative qualifications in Section 5.5 and Appendix H are appropriate; the concern is not that music must establish a universal law. Rather, I would welcome a clearer account of what the framework predicts or explains beyond the reported case studies, or a narrower framing as an empirical methodology for evaluating token interfaces.

  2. The pitch experiments do not fully distinguish relational commitment from input parameterization. Appendix F compares learned categorical embeddings in H with fixed circular coordinates in I, while Appendix A records different active parameter counts. The observed penalty could reflect the fixed input parameterization or its optimization properties, rather than the proposed need to defer relational organization to context. The chromatic arm E is a useful alternative geometry, although D/E also changes pitch factorization. A matched adapter control or an analysis of when the fixed geometry fails would help distinguish these explanations. Appendix G demonstrates context sensitivity, but, as the authors acknowledge, does not directly test this mechanism.

  3. Some central interventions cannot be reconstructed from the submitted description. Appendices A/I do not specify the BPE vocabulary size or merge budget and permitted merge boundaries. Appendices C/F also leave the “music21 consensus high-confidence subset” selection rule unspecified. These choices determine the carrier transformation and canonicalization population, so they matter to interpreting the results. The extensive budget and accounting details are appreciated, but references to frozen repository receipts do not replace these definitions in the available manuscript.

I recommend rejection in the current form. The research question and empirical observations have value, but the framework's explanatory contribution and the interpretation of its central pitch experiment are not yet sufficiently established.

Questions for authors
  1. Scope of the conclusions. Given the model-relative qualifications in Section 5.5 and Appendix H, which findings are specific to symbolic music, and which principles are intended to transfer to natural language or other modalities? What assumptions justify that transfer, and would a more explicitly scoped title or framing better reflect the evidence currently available?

  2. Reconstructing the experimental interventions. Could you specify the BPE vocabulary size, merge budget, and permitted merge boundaries for J/K, together with the selection rule for the “music21 consensus high-confidence subset” used in F/G?

  3. Evidence for the relational mechanism. What evidence distinguishes a penalty from fixed relational geometry in H/I from one caused by the change from learned categorical embeddings to fixed circular inputs and their associated optimization properties?

Reviewer_ibNUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper experiments with several variants of symbolic music tokenization for generation, including different ways to encode time and pitch, with or without music knowledge or relations. The authors compare predictive codelength across these variants and reach conclusions about general principles for tokenization.

Critical strengths

None

Critical weaknesses
  1. The paper is difficult to follow. It uses many abstract terms and lacks clear explanation and organization. It may be AI-readable, but I doubt it is human-readable.
  2. Many terms are defined in the abstract, and the proposed theory has limited connection to the actual experiments. The experiments only provide a very simple specification of the proposed theory, and it could be problematic when tested on other datasets, models, music textures, and domains.
  3. The empirical findings are also limited. Existing tokenization studies explore more ways to encode music and measure results in terms of musicality from more dimensions than predictive codelength.
Questions for authors

I would appreciate it if the authors could address the points in the weakness section.

Reviewer_DdAbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how much information should be encoded during tokenisation and what should be left for the model to learn from context. The proposed Effectiveness–Losslessness Framework determines the boundaries of tokenization based on predictive codelength. Through experiments on symbolic music, they show that explicitly encoding temporal and pitch information can improve prediction, while fixed relational representations and BPE compression do not necessarily lead to better performance.

Critical strengths
  1. An interesting perspective on tokenisation by exploring what information should be encoded in tokens and what should be learned from context.
  2. Clear motivation with the proposed framewrok and provided useful insights into tokenisation design.
Critical weaknesses
  1. Evaluation is somewhat limited, where experiments are conducted under a small set of controlled settings, mainly comparing different tokenisation variants designed by the authors themselves.

  2. I think the paper lacks systematic comparisons with existing mainstream tokenization methods, and the experiments are done using a relatively small Transformer, without explloring larger scale training or evaluation across different model sizes and domains.

Questions for authors

Why the experiment doesn't consider comparing against existing mainstream tokenisation methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.