Compiled Tokenizers: Training Language Models on Schema-Specific Action Spaces

Constrained decoding is commonly used to make language models generate valid structured output, such as SQL, by masking inadmissible tokens at inference time. We introduce Type-Constrained Tokenization (TCT), which compiles the database schema into the tokenizer and training target. The resulting representation lets the model select tables and columns from a candidate list that reflects the current query scope, while the decoder supplies decisions with only one admissible option. Since invalid schema references have no encoding, the model learns to choose among the references and structural alternatives that the schema permits. We compare the models on the 87% of test questions that pass the compiler's encoding and rendering checks. On this set, a 1B compiled model produces 98% fewer outputs that fail to execute than the same model fine-tuned on plain SQL, without reducing execution accuracy. On the full test set, counting the 11% of questions outside the grammar as compiled failures still cuts non-executions by two fifths, while the accuracy difference is not significant. A text control trained to write the same schema choices in ordinary text answers 8% more questions correctly than the compiled model, although 5% of its outputs fail to execute, compared with less than 1% for the compiled model. Given the question, schema, candidate SQL, and a preview of each distinct executable result, a separate 8B instruction model, used without further training, chooses among outputs from models trained to emit plain SQL, schema choices as text, or compiled actions. It answers 10% more questions than the best single model. The same judge answers fewer questions with three rehearsal seeds of either text model, and its accuracy degrades when the compiled candidate is removed.