acceptodds
Under review as a conference paper at ICLR 2027

Constructive Prediction: The Role of Lexical Code Organization in Language Modeling

53 views

Abstract

Character, byte, and subword language models build text from categorical events that are individually valid but may combine into strings with no corresponding entry in a given lexicon. Thus, the prediction interface does not explicitly distinguish such constructions from those that complete a lexical entry. We introduce *constructive prediction*, which instead represents each lexical entry by a short path over a shared, low-cardinality alphabet. In this case, partial paths are intermediate shared states with many possible continuations, and only a completed path can either resolve to a lexical identity or remain unassigned. We instantiate this representation as **Intuitionistic Language Models** (ILM). We first test Flat ILM, which applies a causal Transformer to the coordinate stream. We then extend this design with Full ILM, which adds coordinate-role interfaces and prefix-aware supervision. To assess the effect of constructive prediction, our central experiment asks whether a lexical-to-path assignment constructed from external pretrained embeddings and frozen before downstream language-model training carries predictive structure. To do so, we reassign entries among the same set of paths under the Flat model. Using three assignment seeds, each evaluated with three training seeds, unrestricted reassignment worsens test BPB by on Tiny Shakespeare and on enwik8. All nine paired comparisons favor the original assignment on each corpus. To control for permutation-induced frequency shifts, we use a stricter control that shuffles only entries with equal training frequency, thereby preserving training-weighted coordinate frequencies at each path position. Reassignment still worsens BPB by and , respectively. Beyond assignment, we ask whether the Transformer can exploit path structure. We find that Full ILM has lower mean BPB than Flat ILM at 6.5M and 15.5M on both corpora, and at 100M on enwik8. Full's advantage remains substantial on enwik8 but narrows to BPB at 15.5M on Tiny Shakespeare. Overall, these results show that the organization of lexical identities over shared sequential codes is a modeling variable that can improve prediction under a compact representation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
autoregressive language modeling, constructive prediction, lexical representations, discrete codes, tokenization, pretrained embeddings, compositional representations

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6zk7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Intuitionistic Language Models (ILM), which represent lexical entries as sequences of discrete codes derived from pretrained embeddings. The authors introduce Flat ILM, which predicts these codes autoregressively, and Full ILM, which additionally incorporates role-specific interfaces and prefix-aware supervision. Experiments on Tiny Shakespeare and enwik8 compare the proposed models against character/byte-level and atomic lexical baselines, along with permutation-based controls. The results suggest that embedding-derived code assignments improve predictive performance over randomized assignments, while Full ILM generally achieves lower BPB than Flat ILM.

Critical strengths

The paper provides empirical results across multiple model sizes and random seeds, including permutation-based controls.

Critical weaknesses

Unfair baseline comparisons. The proposed method uses pretrained text-embedding-3-small embeddings, while the Character/Byte GPT baselines do not receive comparable pretrained information. Given the limited training budget, the improvements cannot be clearly attributed to the proposed method. Standard subword baselines are also missing.

Test-set contamination. The vocabulary and code assignments are constructed using the complete corpus, including the test set. This introduces test-set information into representation construction and undermines the standard held-out evaluation setting.

Missing ablations and insufficient justification. The method combines numerous design choices, including PCA, residual clustering, collision repair, role-specific interfaces, and prefix-aware supervision. Their individual contributions are not evaluated, and the motivation for this complexity remains unclear.

Limited generalizability. Experiments are restricted to small datasets, extremely short contexts (approximately 72 bytes), and only 6,000 training updates. Such settings provide little evidence that the findings generalize to modern LLM pretraining.

Questions for authors

How can the authors disentangle the benefits of pretrained embeddings from those of the proposed method, given the unequal baseline conditions?

How would the results change if the vocabulary and code assignments were constructed exclusively from training data?

Can the authors provide ablations to justify the individual components of the proposed framework?

What evidence suggests that the findings would generalize to larger-scale language modeling with longer contexts and sufficient training?

1 comment
AuthorsOpenReview ↗

Response to 6zk7: controlled assignment and added tests

Thank you for pressing on what causes the result and how far it may generalize. We now distinguish the experiment that changes only lexical assignment from comparisons between different model and tokenizer setups. Below, Residual names the submitted residual-clustering map and Recursive names a new map that clusters entries sharing the coordinates already chosen for their paths, both using the same embeddings.

Pretrained embeddings and baselines. You are right that Character/Byte GPT and BPE do not use the pretrained embeddings from which we build our maps. Their BPB values therefore compare complete model and tokenizer setups rather than isolating the effect of those embeddings. Nevertheless, the new byte-level BPE model, fitted on the training split, provides a familiar subword reference. We chose its model width and context limit to make its parameter count and average source-text span close to those of the coordinate models. At 6.5M, we trained BPE and each map's Flat and Full models under both compact optimizer settings for 6,000 and 12,000 updates (Section 6.4 and Appendix C.5).

With the occupied path set, Flat model, and optimizer held fixed, changing which entry receives a path changes held-out BPB. More specifically, exact-frequency reassignment permutes paths only among entries that occur equally often in training, preserving each path's training count. Across both corpora and maps, all 36 pairs favor the original assignment, with three reassignment seeds and three model-training seeds in each setting (Section 6.3, Table 1). Both sides of each pair use paths built from the same pretrained embeddings. This isolates the assignment within a shared embedding source, while leaving open whether pretrained embeddings are necessary for an assignment effect.

Held-out information. The path map includes held-out lexical identities, so the representation protocol is transductive. Under our fixed lexical-entry path formulation, scoring the full test stream requires paths for those entries as well. Those entries receive external embeddings, and the construction does not use their held-out occurrence counts or prediction errors. Only training sequences update the Transformer (Section 5). On enwik8, entries absent from training account for 3.603.60% of scored test bytes. However, that share alone does not show whether those entries drive the assignment penalty. We therefore also scored targets whose entries appeared in training. Even on these targets, the Residual exact-frequency penalty is +0.0950±0.0029+0.0950\pm0.0029 BPB, and Recursive gives +0.0812±0.0038+0.0812\pm0.0038 (Appendix D.5).

These scores still use maps constructed with the full lexical inventory, so they do not tell us what a train-only map would produce. Under our fixed lexical-entry path formulation, such a map would leave test-only entries without paths. Scoring the same full test would require a lossless way to encode them, and we have not evaluated that variant.

Construction and Full ablations. The assignment penalty persists under Recursive, which omits Residual's subtraction and collision repair. Both maps use the same pretrained embeddings and PCA-reduced vectors, while Recursive splits entries by their current prefix. Recursive also reaches lower test BPB (Section 6.1 and Appendix A). Thus the comparison tests the complete constructions after their common embedding and PCA steps.

The input and output role interfaces account for most of Full's enwik8 gain at 6.5M. We trained variants that distinguish those two interfaces from the training mask. The Residual exact-frequency penalty also remains when the interfaces tell the model where a coordinate lies within its path (Section 6.2 and Appendices C.4 and D.6).

More updates and capacity. At 100M and 12,000 updates on enwik8, the assignment penalty and Full's lower BPB both persist. The 6,000-update experiments left this question open, so we trained the 100M enwik8 models to 12,000 updates under a shared cosine schedule. The paired BPB differences below are mean ±\pm SD across three model-training seeds:

100M enwik8 BPB difference at 12kResidualRecursive
Exact-frequency Permuted Flat minus Flat+0.1099±0.0038+0.1099\pm0.0038+0.1036±0.0040+0.1036\pm0.0040
Flat minus Full+0.0811±0.0024+0.0811\pm0.0024+0.0710±0.0029+0.0710\pm0.0029

Positive gaps favor original Flat in the first row and Full in the second. Each Exact comparison uses one assignment seed per map. Under the same regime, Recursive Full scores 1.9579±0.00131.9579\pm0.0013 BPB versus BPE's 2.1951±0.00212.1951\pm0.0021 as a complete-system reference (Section 6.5 and Appendix C.6). The paper frames these as short-context results and leaves modern long-context pretraining for future study.

We would be grateful if you reconsidered your assessment in light of these additional controls and the more precisely scoped result.

Reviewer_cSkhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper analyzes constructive prediction, which represents each lexical entry as a fixed length path over a shared alphabet. A deterministic lexical scanner defines the entries, and external pretrained embeddings, PCA, residual clustering, and deterministic collision repair produce the lexical-to-code map. Flat ILM applies a standard causal Transformer to the flattened coordinate stream. Full ILM retains the same paths and Transformer backbone while adding role specific input and output interfaces and prefix aware supervision. A coordinate role identifies the position of a coordinate within its lexical path.

Experiments compare the original Flat ILM with Permuted Flat, which reassigns lexical entries among the same occupied paths. The exact-frequency variant permits reassignment only between entries with equal training counts, preserving complete path counts and within-path prefix counts. Both permutation controls increase teacher forced test BPB on Tiny Shakespeare and enwik8. Supporting experiments compare Flat with Atomic Lexical and Character/Byte GPT, while the Flat–Full comparison extends to 100M parameters on enwik8. The study evaluates a closed corpus representation whose vocabulary and final map are constructed from the complete frozen corpus.

Critical strengths

The main strength of this paper is its careful evaluation of the lexical-to-code assignment. The Permuted Flat experiments preserve the lexical segmentation, occupied paths, alphabet size, path length, model architecture, and optimization settings while changing the association between lexical entries and paths. The exact-frequency control further preserves complete path counts, prefix counts, and training weighted coordinate frequencies. The reassignment increases BPB by 0.1063 on Tiny Shakespeare and 0.0957 on enwik8, with all nine paired comparisons favoring the original assignment on both corpora. These results provide convincing evidence that the final lexical-to-code assignment carries predictive structure.

Critical weaknesses

Weakness 1: The experiments do not sufficiently explain the improvements from the original lexical-to-code assignment and Full ILM. The vocabulary and final map are constructed using the frozen corpus, including held out lexical entries. As a result, entries absent from training still receive paths constructed from external pretrained embeddings, and exact-frequency reassignment can change these paths. However, the paper does not report how much of the BPB improvement comes from entries absent from training compared with entries observed during training. This is important because the models are trained for only 6,000 updates with relatively short contexts, and the main permutation controls are evaluated only at 6.5M parameters. The supporting comparisons also do not fully explain the improvement. Atomic Lexical differs in parameter allocation and does not use the fixed embedding derived assignment, while the Character/Byte GPT references retain architectural differences even after the optimizer crossover. Furthermore, Full ILM introduces role specific interfaces and prefix aware supervision together, without evaluating their individual contributions. These limitations make it difficult to determine which parts of the proposed approach are responsible for the reported results.

Weakness 2: The novelty relative to previous work is limited, and the motivating problem is not directly evaluated. Previous works have already investigated how the organization of words in a hierarchical output vocabulary affects language model prediction. Mnih and Hinton (2008) constructed word trees from learned word representations and showed that these trees achieve lower perplexity than random trees. Similarly, Le et al. (2011) used PCA and recursive k-means clustering to organize the output vocabulary of a neural language model. LightRNN (Li et al., 2016) and Hybrid-LightRNN (Kong et al., 2019) also predict lexical entries through shared code components, while DSI (Tay et al., 2022) and TIGER (Rajput et al., 2023) investigate structured identifiers for documents and items, respectively. Although the proposed fixed embedding derived lexical-to-code map and exact-frequency permutation control distinguish this paper from previous work, the experiments do not sufficiently explain which properties of the coordinate construction lead to better prediction. Furthermore, the paper motivates constructive prediction by distinguishing completed lexical paths from unassigned paths, but its decoder can still complete unassigned paths. Since the experiments evaluate teacher forced BPB, they do not directly examine this aspect of constructive prediction.

Questions for authors

Q1. Could the authors report the BPB differences between the original assignment and exact-frequency Permuted Flat separately for lexical entries with zero, low, and high training frequencies? In particular, does the original assignment consistently achieve lower BPB across these groups? This analysis would help clarify which lexical entries contribute most to the improvement from the original lexical-to-code assignment.

Q2. Could the authors further clarify what distinguishes the proposed embedding derived lexical-to-code assignment from previous approaches that organize word representations through clustering? In particular, would a simpler assignment constructed from the same external pretrained embeddings achieve similar BPB? This comparison would help clarify the contribution of the proposed coordinate construction beyond existing approaches.

Q3. Could the authors provide separate ablation experiments for Full ILM's role specific interfaces and prefix aware supervision? It would also be useful to repeat the exact-frequency permutation experiment under Full ILM or Flat ILM with explicit coordinate role information. These experiments would help determine which components contribute to Full ILM's improvement and whether the benefit of the original lexical-to-code assignment persists when coordinate roles are explicitly represented.

1 comment
AuthorsOpenReview ↗

Response to cSkh: frequency groups, simpler map, and roles

Thank you for the specific questions. They helped us design follow-ups that separate how often an entry was seen, how its path was built, and which parts of Full matter. Here Residual refers to the submitted residual-clustering map, while Recursive refers to a new map built from the same embeddings.

Because Atomic Lexical changes parameter allocation and Character/Byte GPT uses a different Transformer implementation, we treat them as supporting references. By contrast, the paired Flat runs provide the controlled assignment test.

Q1. Does the assignment penalty persist across entries with zero, low, and high training counts?

Yes, original Flat has lower BPB than Exact-frequency Permuted Flat in all nine paired runs within every training-count group under both maps and on both corpora. Exact-frequency reassignment permutes paths only among entries with equal training counts. We grouped test targets by those counts, separating entries unseen in training, seen one to five times, and seen at least six times. Each of three reassignments is evaluated with all three model-training seeds, giving nine paired runs per corpus-map setting. The SD is across reassignment means, each averaging three training runs. The within-group penalty below is Permuted minus original BPB on that group's targets. Its contribution to the whole-test penalty weights that difference by the group's share of scored bytes. The table shows Residual-map enwik8, and Appendix D.5 reports the full analysis.

Training countScored-byte shareWithin-group BPB penaltyContribution to whole-test penalty
003.60%3.60\%+0.1148±0.0406+0.1148\pm0.0406+0.0041±0.0015+0.0041\pm0.0015
11-553.47%3.47\%+0.1471±0.0062+0.1471\pm0.0062+0.0051±0.0002+0.0051\pm0.0002
≥6\geq692.93%92.93\%+0.0931±0.0028+0.0931\pm0.0028+0.0865±0.0026+0.0865\pm0.0026

Entries seen at least six times account for 92.9392.93% of scored bytes and contribute +0.0865+0.0865 BPB to the +0.0957+0.0957 overall penalty. The effect is therefore present even on entries repeatedly observed during training. As in the main comparisons, these scores use the reported full-corpus maps.

Q2. Would a simpler map from the same embeddings retain the effect, and how does this approach differ from prior work?

Yes, we built a simpler map called Recursive from the same external embeddings. It retains the exact-frequency assignment penalty on both corpora and lowers Flat and Full BPB at 6.5M (Sections 6.1 and 6.3, Table 1). Recursive builds paths by splitting groups of entries with the same prefix, without Residual's subtraction or collision repair. At 100M and 12,000 updates on enwik8, the penalty remains +0.1099±0.0038+0.1099\pm0.0038 BPB under Residual and +0.1036±0.0040+0.1036\pm0.0040 under Recursive. Each larger-scale check uses one reassignment and three model-training seeds (Section 6.5). Thus the assignment effect survives both constructions, while the choice of construction also changes test BPB.

A key distinction from this prior work is our paired test of a map derived from external embeddings and frozen before language-model training. Mnih and Hinton, as well as Le et al., factor next-word probabilities through hierarchical output trees, while LightRNN and Hybrid-LightRNN already predict code components successively. Related Work now makes this distinction explicit without attributing hierarchical coding itself to this paper.

Q3. Which parts of Full account for its gain, and does the assignment effect persist with explicit coordinate roles?

On enwik8, the role-specific interfaces account for most of Full's 6.5M gain. Adding the prefix-aware mask after both interfaces has little further BPB effect. More specifically, we trained variants that separate Full's role-specific input table, output head, and training mask (Section 6.2 and Appendix C.4). The input table reads a coordinate according to its position in a path, while the output head predicts according to the next position. The mask selects which positions contribute to training loss.

The assignment penalty also remains when the model knows the coordinate role. Under ordinary unmasked training, we repeated the Residual exact-frequency comparison with role-specific input, output, and combined interfaces. On enwik8, the paired penalties are +0.0934±0.0043+0.0934\pm0.0043 BPB with input roles, +0.0931±0.0026+0.0931\pm0.0026 with output roles, and +0.1044±0.0057+0.1044\pm0.0057 with both (Appendix D.6). Each condition uses one assignment seed and three model-training seeds.

Completed versus unassigned paths. An unconstrained decoder could complete an unassigned path. Our BPB experiments score observed paths assigned to lexical entries, so they do not test that generation behavior. The revised Introduction describes an occupied-path constraint as a possible generation rule, which we have not evaluated.

We would appreciate your reassessment in light of these added tests and the revised novelty framing.

Reviewer_qWutOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes constructive prediction, in which each lexical entry is mapped to a fixed-length path of coordinates over a small alphabet and the language model predicts these coordinates sequentially. To build this map, PCA, residual spherical k-means and collision repair are applied to text-embedding-3-small embeddings of the lexicon. Once computed, the map stays frozen throughout training. Building on this representation, the paper studies two variants. Flat ILM is a standard causal Transformer trained on the coordinate stream, and Full ILM extends Flat ILM with coordinate-role interfaces and a prefix-aware loss mask. In the main experiment, shuffling the lexicon-to-path map, whether freely or within classes of equal training frequency, consistently increases BPB. On Tiny Shakespeare and enwik8, further experiments compare against a word-level model and char/byte-level GPT baselines, and show that Full ILM outperforms Flat ILM at 6.5M, 15.5M and 100M parameters.

Critical strengths
  1. The paper offers a principled way to factorize a large vocabulary into short paths over a small shared alphabet, turning one vocabulary-sized prediction into several low-cardinality decisions.

  2. The frozen, embedding-derived map cleanly separates lexical organization from language-model training, providing a controlled setting to test whether externally derived structure supports prediction.

  3. The design of Full ILM is well motivated, as its role-specific interfaces and prefix-aware loss directly address position-dependent coordinate meanings and windows that start mid-path, without changing causal inference.

Critical weaknesses
  1. The lexicon is built from the full corpus, including the test split, so words that appear only at test time still receive informative paths. This leaks test-set information into the lexical map and makes the comparison with char/byte baselines unfair.

  2. The char/byte baselines appear under-tuned. On Tiny Shakespeare, Character GPT reaches only 2.65 to 2.72 BPB, whereas the nanoGPT result cited in Appendix C.5 is 2.120.

  3. All models are trained for only 6,000 updates, which is less than one epoch of enwik8. This budget is likely insufficient for the larger models, especially the 100M one, so it is unclear whether the conclusions hold under adequate training.

  4. The gain of Full ILM over Flat ILM may come mainly from explicitly providing the coordinate role, which Flat ILM must infer from context. The role interfaces and the prefix-aware mask are not ablated separately, so the source of the improvement remains unclear.

Questions for authors
  1. How do the results change if the lexicon and the map are built on the training split only? What fraction of test tokens are unseen during training?

  2. Could the authors ablate the role interfaces and the prefix-aware mask in Full ILM separately, to clarify which component drives the improvement?

  3. Could the authors report char/byte GPT results under the standard nanoGPT setting, together with a BPE baseline that matches the parameter count and context length?

1 comment
AuthorsOpenReview ↗

Response to qWut: coverage, baselines, horizon, and Full

Thank you for laying out the concerns about unseen test entries, the reference models, training length, and Full's components. We investigated them separately because each calls for a different comparison. Below, Residual names the submitted residual-clustering map, while Recursive names a new map that clusters entries according to the coordinates already chosen for their paths.

Full-corpus map and unseen test entries. The map is transductive because held-out lexical identities participate in the frozen representation, while only training sequences update the Transformer (Section 5). Under our fixed-path design, scoring the full held-out text requires a path for every lexical entry, including entries absent from training. Those entries are included in the frozen vocabulary and receive external embeddings. The construction does not use their held-out occurrence counts or prediction errors. Those test-only entries account for 7.407.40% of test lexical-entry occurrences on Tiny Shakespeare and 1.671.67% on enwik8, or 15.5115.51% and 3.603.60% of scored test bytes, respectively (Appendix D.5).

Exact-frequency reassignment permutes paths only among entries appearing equally often in training. When we rescore only targets whose entries occurred in training, paired penalties remain 0.0592±0.00270.0592\pm0.0027 BPB on Tiny Shakespeare and 0.0950±0.00290.0950\pm0.0029 on enwik8 (Appendix D.5). This check still uses the full-corpus map, where held-out entries can affect paths assigned to training entries. By contrast, a train-only map under our fixed-path design would leave test-only entries without paths. Scoring the same full test would require a lossless way to encode them, and we have not evaluated that variant.

Raw-symbol and BPE references. The external 2.1202.120 Tiny Shakespeare nanoGPT result uses a different data split, so it cannot be compared directly with our test BPB (Appendix C.7). Because of that split difference, we trained Character/Byte GPT and Residual Flat on our own splits under both the ILM and nanoGPT optimizer settings. Under the nanoGPT setting, Character GPT scores 2.7167±0.01682.7167\pm0.0168 BPB on Tiny Shakespeare, while Byte GPT scores 2.6401±0.00392.6401\pm0.0039 on enwik8 (Appendix C.3).

We also added the subword comparison you requested, fitting a 4,096-token byte-level BPE vocabulary on the enwik8 training split. We chose its model width and context limit to make its parameter count and average source-text span close to those of the coordinate models. At 6.5M, we trained BPE and each map's Flat and Full models under both optimizer settings for 6,000 and 12,000 updates. Under the nanoGPT setting at 6,000 updates, for example, BPE scores 2.2563±0.01312.2563\pm0.0131 BPB and Recursive Full scores 2.0852±0.00242.0852\pm0.0024 (Section 6.4 and Appendix C.5). Section 6.5 extends the BPE reference to 100M. Because Character/Byte GPT and BPE do not use ILM's external embeddings or full-corpus lexical map, these BPB scores compare complete systems with different tokenizations and implementations. The reassignment experiment instead holds the Flat model fixed to test the lexical assignment.

Horizon and components. The assignment penalty and Full's lower BPB both persist after 12,000 updates on enwik8. At 100M, Exact-frequency Permuted Flat exceeds original Flat by 0.1099±0.00380.1099\pm0.0038 BPB under Residual and 0.1036±0.00400.1036\pm0.0040 under Recursive. Each result uses one assignment seed and three paired model-training seeds. Full remains below Flat by 0.0811±0.00240.0811\pm0.0024 and 0.0710±0.00290.0710\pm0.0029, respectively (Section 6.5 and Appendix C.6).

At 6.5M on enwik8, we tested Full's role-specific input table, output head, and training mask separately. The two interfaces account for most of Full's gain, while the mask adds little once they are combined. Mask only adds the mask to Flat, while Input only and Output only each add one role interface. Both roles combines the interfaces without the mask, and Full adds the mask. The table gives mean ±\pm SD BPB over three model-training seeds (Section 6.2 and Appendix C.4):

6.5M enwik8 conditionResidualRecursive
Flat2.3331±0.00452.3331\pm0.00452.1891±0.00442.1891\pm0.0044
Mask only2.3221±0.00282.3221\pm0.00282.1765±0.00352.1765\pm0.0035
Input only2.2637±0.00372.2637\pm0.00372.1366±0.00122.1366\pm0.0012
Output only2.2804±0.00242.2804\pm0.00242.1317±0.00502.1317\pm0.0050
Both roles2.2344±0.00642.2344\pm0.00642.1062±0.00702.1062\pm0.0070
Full2.2359±0.00632.2359\pm0.00632.1081±0.00452.1081\pm0.0045

Lower is better. Under both maps, either role interface helps, while Both roles nearly matches Full. Because Both roles and Full differ only in whether the prefix-aware training mask is applied, their near-equality shows that the mask has little further BPB effect at this tier.

We would be grateful if you reconsidered the soundness assessment and recommendation in light of these results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.