- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Intuitionistic Language Models (ILM), which represent lexical entries as sequences of discrete codes derived from pretrained embeddings. The authors introduce Flat ILM, which predicts these codes autoregressively, and Full ILM, which additionally incorporates role-specific interfaces and prefix-aware supervision. Experiments on Tiny Shakespeare and enwik8 compare the proposed models against character/byte-level and atomic lexical baselines, along with permutation-based controls. The results suggest that embedding-derived code assignments improve predictive performance over randomized assignments, while Full ILM generally achieves lower BPB than Flat ILM.
Critical strengths
The paper provides empirical results across multiple model sizes and random seeds, including permutation-based controls.
Critical weaknesses
Unfair baseline comparisons. The proposed method uses pretrained text-embedding-3-small embeddings, while the Character/Byte GPT baselines do not receive comparable pretrained information. Given the limited training budget, the improvements cannot be clearly attributed to the proposed method. Standard subword baselines are also missing.
Test-set contamination. The vocabulary and code assignments are constructed using the complete corpus, including the test set. This introduces test-set information into representation construction and undermines the standard held-out evaluation setting.
Missing ablations and insufficient justification. The method combines numerous design choices, including PCA, residual clustering, collision repair, role-specific interfaces, and prefix-aware supervision. Their individual contributions are not evaluated, and the motivation for this complexity remains unclear.
Limited generalizability. Experiments are restricted to small datasets, extremely short contexts (approximately 72 bytes), and only 6,000 training updates. Such settings provide little evidence that the findings generalize to modern LLM pretraining.
Questions for authors
How can the authors disentangle the benefits of pretrained embeddings from those of the proposed method, given the unequal baseline conditions?
How would the results change if the vocabulary and code assignments were constructed exclusively from training data?
Can the authors provide ablations to justify the individual components of the proposed framework?
What evidence suggests that the findings would generalize to larger-scale language modeling with longer contexts and sufficient training?
Response to 6zk7: controlled assignment and added tests
Thank you for pressing on what causes the result and how far it may generalize. We now distinguish the experiment that changes only lexical assignment from comparisons between different model and tokenizer setups. Below, Residual names the submitted residual-clustering map and Recursive names a new map that clusters entries sharing the coordinates already chosen for their paths, both using the same embeddings.
Pretrained embeddings and baselines. You are right that Character/Byte GPT and BPE do not use the pretrained embeddings from which we build our maps. Their BPB values therefore compare complete model and tokenizer setups rather than isolating the effect of those embeddings. Nevertheless, the new byte-level BPE model, fitted on the training split, provides a familiar subword reference. We chose its model width and context limit to make its parameter count and average source-text span close to those of the coordinate models. At 6.5M, we trained BPE and each map's Flat and Full models under both compact optimizer settings for 6,000 and 12,000 updates (Section 6.4 and Appendix C.5).
With the occupied path set, Flat model, and optimizer held fixed, changing which entry receives a path changes held-out BPB. More specifically, exact-frequency reassignment permutes paths only among entries that occur equally often in training, preserving each path's training count. Across both corpora and maps, all 36 pairs favor the original assignment, with three reassignment seeds and three model-training seeds in each setting (Section 6.3, Table 1). Both sides of each pair use paths built from the same pretrained embeddings. This isolates the assignment within a shared embedding source, while leaving open whether pretrained embeddings are necessary for an assignment effect.
Held-out information. The path map includes held-out lexical identities, so the representation protocol is transductive. Under our fixed lexical-entry path formulation, scoring the full test stream requires paths for those entries as well. Those entries receive external embeddings, and the construction does not use their held-out occurrence counts or prediction errors. Only training sequences update the Transformer (Section 5). On enwik8, entries absent from training account for 3.60% of scored test bytes. However, that share alone does not show whether those entries drive the assignment penalty. We therefore also scored targets whose entries appeared in training. Even on these targets, the Residual exact-frequency penalty is +0.0950±0.0029 BPB, and Recursive gives +0.0812±0.0038 (Appendix D.5).
These scores still use maps constructed with the full lexical inventory, so they do not tell us what a train-only map would produce. Under our fixed lexical-entry path formulation, such a map would leave test-only entries without paths. Scoring the same full test would require a lossless way to encode them, and we have not evaluated that variant.
Construction and Full ablations. The assignment penalty persists under Recursive, which omits Residual's subtraction and collision repair. Both maps use the same pretrained embeddings and PCA-reduced vectors, while Recursive splits entries by their current prefix. Recursive also reaches lower test BPB (Section 6.1 and Appendix A). Thus the comparison tests the complete constructions after their common embedding and PCA steps.
The input and output role interfaces account for most of Full's enwik8 gain at 6.5M. We trained variants that distinguish those two interfaces from the training mask. The Residual exact-frequency penalty also remains when the interfaces tell the model where a coordinate lies within its path (Section 6.2 and Appendices C.4 and D.6).
More updates and capacity. At 100M and 12,000 updates on enwik8, the assignment penalty and Full's lower BPB both persist. The 6,000-update experiments left this question open, so we trained the 100M enwik8 models to 12,000 updates under a shared cosine schedule. The paired BPB differences below are mean ± SD across three model-training seeds:
Positive gaps favor original Flat in the first row and Full in the second. Each Exact comparison uses one assignment seed per map. Under the same regime, Recursive Full scores 1.9579±0.0013 BPB versus BPE's 2.1951±0.0021 as a complete-system reference (Section 6.5 and Appendix C.6). The paper frames these as short-context results and leaves modern long-context pretraining for future study.
We would be grateful if you reconsidered your assessment in light of these additional controls and the more precisely scoped result.