- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper uses communication complexity to study how token-boundary placement affects shallow Transformer width. It derives boundary-load formulas for fixed-chunk tokenizers, constructs a matched tokenizer family , and reports one-layer/two-layer experiments on complementary intersection tasks. Empirically, one-layer models fail under misaligned tokenizers at , while depth-two repair does not hold at .
Critical strengths
- Well-controlled paired experiments. Sharing data, initialization, parameters, and optimization isolates boundary placement as the intervention.
- Novel theoretical lens. Communication complexity gives a fresh formal angle on tokenizer-induced expressivity costs.
Critical weaknesses
-
Theory–experiment mismatch. At , width 128 is far above the one-layer lower-bound scale, so chance accuracy is better read as a trainability failure than width evidence. The exact two-layer construction gives an -width solution, yet learned depth-two models stay at chance at . The authors acknowledge this but give no mechanism. This undercuts the learned-model narrative.
-
The metric is task-dependent. Moving a boundary transfers load between complementary tasks, so boundary load cannot rank tokenizers globally like vocabulary size or compression. The authors acknowledge this, but the broad framing still suggests a general account of token boundaries and width. Acknowledgment does not resolve the practical-value gap.
-
Scope is too narrow for the claims. Exact results require full product domains, fixed exact-substring chunks, deterministic fixed-length tokenization, and exact finite precision. Proposition 1 weakens with large state counts; the BPE corollary covers only two synthetic endpoints. The main lower-bound engine is prior work, so the contribution is a specialization rather than a new principle.
-
Presentation obscures the contribution. The main text is dense, lacks a glossary and intuitive explanation of , and depends heavily on a long appendix; the 8.5-page body does not stand alone. Related Work relies on repeated “We do not claim…” positioning, making the positive novelty hard to identify. This is a clarity and positioning problem, not merely a length issue.
Questions for authors
- At , width 128 is above the one-layer lower-bound scale. What evidence supports a width interpretation rather than a trainability failure?
- Depth-two stays at chance at despite the exact -width construction. Please report diagnostics—training accuracy, loss, gradients, learning-rate sensitivity, Track B results—to distinguish optimization failure from finite-width or implementation mismatch.
- Given the task-dependent metric and narrow theoretical scope, what practical conclusion should the NLP community draw? Can you provide a boundary-load profile for a deployed tokenizer beyond the digit-grouping correspondence?