acceptodds
Withdrawn from ICLR 2027

Compiled Tokenizers: Training Language Models on Schema-Specific Action Spaces

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Constrained decoding is commonly used to make language models generate valid structured output, such as SQL, by masking inadmissible tokens at inference time. We introduce Type-Constrained Tokenization (TCT), which compiles the database schema into the tokenizer and training target. The resulting representation lets the model select tables and columns from a candidate list that reflects the current query scope, while the decoder supplies decisions with only one admissible option. Since invalid schema references have no encoding, the model learns to choose among the references and structural alternatives that the schema permits. We compare the models on the 87% of test questions that pass the compiler's encoding and rendering checks. On this set, a 1B compiled model produces 98% fewer outputs that fail to execute than the same model fine-tuned on plain SQL, without reducing execution accuracy. On the full test set, counting the 11% of questions outside the grammar as compiled failures still cuts non-executions by two fifths, while the accuracy difference is not significant. A text control trained to write the same schema choices in ordinary text answers 8% more questions correctly than the compiled model, although 5% of its outputs fail to execute, compared with less than 1% for the compiled model. Given the question, schema, candidate SQL, and a preview of each distinct executable result, a separate 8B instruction model, used without further training, chooses among outputs from models trained to emit plain SQL, schema choices as text, or compiled actions. It answers 10% more questions than the best single model. The same judge answers fewer questions with three rehearsal seeds of either text model, and its accuracy degrades when the compiled candidate is removed.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
tokenization, constrained decoding, structured generation, text-to-SQL, database schemas, language models

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_SCvPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Type-Constrained Tokenization (TCT), which compiles database schemas into scope-aware menus of table and column choices and converts SQL into action sequences for model training. The system automatically supplies forced actions, filters inadmissible choices during generation, and renders the output as SQL. The authors primarily compare plain SQL, menu-index text, and compiled actions on Spider, with additional ablations, perturbation tests, and candidate-selection experiments. TCT substantially reduces execution failures, but an independent accuracy advantage from its tokenization has not been established. Candidates generated using different representations also exhibit some complementarity.

Critical strengths
  • Clear improvement in execution reliability. On the common set of 1,863 test questions, TCT produces 6 execution failures, compared with 324 for plain SQL. Even when unsupported questions are counted as failures, the full-test-set failure count decreases from 405 to 245. These results demonstrate that the system effectively reduces invalid queries.

  • The representation accommodates unseen database schemas. TCT represents table and column references using reusable menu indices. New databases can use the same model by rebinding the menus, without adding dedicated tokens for every new table or column name. Evaluation on 40 unseen databases provides a concrete demonstration of integrating database constraints with a pretrained model.

Critical weaknesses
  • The independent contribution of the specialized tokenizer is not sufficiently isolated. The paper lacks a direct comparison against menu-index text with constraints equivalent to TCT’s. The mask ablation demonstrates the importance of inference-time filtering, but does not establish the additional benefit of specialized tokenization over constrained text representations. The earlier constrained-decoding experiment in §5.5 is also not a matched comparison, leaving the central contribution insufficiently isolated.

  • Training data and checkpoint-selection budgets are not fully matched. Plain SQL uses 6,106 second-stage training examples, whereas TCT and menu-index text use 5,335. TCT selects the best development checkpoint among five rehearsal seeds, while the plain-SQL anchor is fixed beforehand. Matching optimization steps does not eliminate these differences, making it difficult to attribute accuracy changes solely to the output representation.

  • The absence of an execution-feedback repair baseline leaves TCT’s practical value uncertain. SQL agents (e.g., MAC-SQL) can already repair queries using database error messages. It therefore remains unclear whether TCT offers substantial performance gains within current agent-based workflows.

  • Grammar coverage and evaluation scope limit the applicability of the findings. The implementation supports constructs such as filtering, joins, aggregation, and uncorrelated subqueries, but excludes self-joins, set operations, and correlated subqueries. The main three-model comparison covers only 1,863 of 2,147 test questions, and the evaluation primarily uses Spider and its perturbation suites. Evidence for more complex SQL and other data distributions therefore remains limited.

  • Insufficient detail about training-data construction limits reproducibility. The paper uses schema-literacy, Spider, and rehearsal stages, but does not sufficiently describe the data sources, construction templates, sample counts, or cross-stage overlap for the first and third stages. The description is insufficient to reconstruct the full training pipeline, and the contribution of each stage is not independently analyzed (§4, Appendix B).

Questions for authors
  • Does the specialized tokenizer provide benefits beyond constraint filtering? Menu-index text could also be decoded under equivalent grammar and schema constraints. Could you compare this constrained text baseline with TCT under matched conditions? This comparison would help determine whether TCT’s gains come from its specialized tokenization or from inference-time constraints.

  • Do the main findings hold under matched training and selection conditions? Could you provide comparisons using identical training examples and development-set selection budgets, together with independent repetitions of the full training pipeline? The existing rehearsal-seed controls do not capture variability across the entire training process.

  • What advantages remain over execution-feedback repair? How do plain SQL and menu-index text perform after receiving specific database error messages and being allowed one or two repair rounds? Please also report generation volume, database calls, or measured latency to assess the performance–cost tradeoff.

  • How would the system handle unsupported queries in deployment? Without access to reference SQL, how does the system handle requests beyond its grammar? Could you evaluate whether the fallback or judge gains persist on the full test set using a routing procedure that does not rely on reference answers?

  • Could you provide the complete data-construction procedure? Please specify the data sources, sample counts, construction and filtering rules, and database- and example-level overlap across the three stages. A complete example showing how the same input is converted into all three training targets would also be helpful.

  • How robust are the accuracy findings under a consistent scoring protocol? Building on the reported sensitivity analyses, could you summarize results on the same test set using scoring that respects ORDER BY, and provide database-clustered confidence intervals for the main accuracy differences? This would clarify which comparisons support accuracy gains and which primarily support reductions in execution failures.

Reviewer_8n4GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Type-Constrained Tokenization (TCT), a framework that incorporates database schema constraints into both the training representation and decoding process for structured language generation. Instead of generating SQL directly as text, the model predicts schema-specific actions using dynamically constructed menus of admissible tables, columns, and structural operations. The framework also omits deterministic decisions and uses constrained decoding to prevent invalid schema references. Experiments on Spider demonstrate a substantial reduction in SQL execution failures compared with plain-SQL generation. However, the accuracy improvements are less conclusive, as a menu-text baseline achieves higher execution accuracy than compiled tokenization. The paper further explores representation diversity by combining plain SQL, menu-text, and compiled outputs through an execution-based verifier, demonstrating improved performance over individual models and several same-representation ensembles.

Critical strengths
  1. The paper integrates schema-aware menus, scope tracking, and compiled action tokens into a unified training and decoding framework.
  2. The menu-text baseline helps distinguish the benefits of structured schema choices from those of specialized compiled tokens.
  3. ombining outputs from different representations with an execution-based verifier improves accuracy compared with individual models and several same-representation ensembles.
Critical weaknesses
  1. The individual effects of compiled tokens, schema-indexed menus, and inference-time grammar constraints are not sufficiently isolated.
  2. The schema-aware constrained-decoding baseline uses different checkpoints and grammar coverage, limiting the strength of the comparison. 3 Training data sizes, output representations, and checkpoint-selection procedures differ across models, making causal attribution difficult.
Questions for authors
  1. Could the authors provide controlled ablations that separately evaluate compiled action tokens, schema-indexed menus, deterministic-action omission, and inference-time grammar masking? This would help clarify which components primarily contribute to the substantial reduction in execution failures.
  2. Could the authors compare TCT with a stronger schema-aware constrained-decoding method under matched training data, checkpoints, grammar coverage, and evaluation protocols?
  3. How sensitive are the results to differences in training data size and checkpoint selection? Could the authors provide matched-data and multi-seed comparisons to better isolate the effect of the output representation?
Reviewer_vtaoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a compiler that converts a database schema and the SQL grammar into a set of action tokens. During training, the compiler encodes each gold query into action tokens and removes the steps with only one possible option. Thus, the model learns to predict only the steps with two or more options. During inference, the compiler gives the allowed options at each step. The decoder masks all other options and writes the forced steps. The output starts with the FROM clause, where the model selects a table from the menu of all tables in the database. At the end, a renderer converts the action tokens into SQL.

Critical strengths

None

Critical weaknesses
  1. Inadequate technical description One of the major contributions of the paper is the compiler that decides which options are available at each decoding step. Yet, how the compiler is built and how it works are not adequately described in the paper.

  2. Unclear practical contribution The paper focuses mainly on improving query execution success rate while this metric is improved the answer accuracy does not really follow. On the common 1,863 test questions out of the 2147 Spider test questions that the grammar supports, execution failures fall from 324 to 6, but correct answers rise only from 1,108 to 1,153 (Table 1). On all 2,147 test questions, the lead over plain SQL is 12 answers (~0.56% improvement). The paper itself also admits that when using a 8B generator model with the proposed compiler the accuracy is reduced (line 465-467)

  3. Limited experimental scope All experiments use Spider 1.0 or perturbed versions of its development set (Dr.Spider, Spider-Syn, Spider-Realistic). The paper does not test more complex and realistic benchmarks such as BIRD (Li et al., 2023) or Spider 2.0 (Lei et al., 2025). The grammar already covers only 88.9% of the Spider test questions (Section 4), so its coverage on these benchmarks is possibly much lower.

Questions for authors
  1. Can the author clarify the motivation on why improved execution success rate without clear accuracy improvement might be a benefit given that the plain SQL baselines can always do regeneration?
  2. Can the author describe how the compiler was built? what operations are supported?
Reviewer_xWBSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Type-Constrained Tokenization (TCT), which turns a database schema and the current query scope into an action space for SQL generation. The model selects tables and columns from allowed menus, while the decoder supplies actions with only one valid choice. The authors compare this approach with plain SQL generation and a text-based menu control on Spider 1.0. They also use a separate 8B judge to select among executed answers. The evaluation covers execution success, answer accuracy, robustness, and candidate-pool controls.

Critical strengths
  1. The compiled model greatly reduces execution failures. On the common set of 1,863 test questions, only 6 compiled outputs fail to execute, compared with 324 plain-SQL outputs. The mask ablation provides further support for the grammar constraint.

  2. The authors report results that help define the limits of their claims. The menu-text control, full-test-set results, and candidate-pool ablations help distinguish execution success from answer accuracy.

Critical weaknesses
  1. Limited grammar coverage weakens the accuracy claim. The grammar does not support 239 of the 2,147 test questions, and a later render check removes another 45 from the three-model comparison. The compiled model answers 48 more questions correctly than plain SQL on the original 1,908 supported questions, but only 12 more on the full test set. The full-set difference is not statistically significant.

  2. The experiments do not fully isolate the benefit of compiled tokens. The plain-SQL model uses 771 more second-stage training examples, and checkpoint selection differs across models. The menu-text control answers more questions correctly than the compiled model on the common test set. Adding the compiled candidate improves the judge’s result, but it also increases the number of questions with at least one correct candidate. The evidence therefore supports the value of complementary candidates, while leaving the effect on the judge’s selection ability unclear.

Questions for authors
  1. It would be helpful to see results for plain SQL, menu text, and compiled output with the same second-stage data, the same checkpoint-selection budget, and several full training seeds. Can the authors provide this comparison?
  2. How would a routing rule that does not use the reference SQL perform on all 2,147 test questions? The authors could also clarify how many of the additional 45 exclusions reflect a semantic mismatch rather than a harmless difference in SQL form.
  3. A comparison with a judge given repeated menu-text samples under similar budgets would help explain the three-model gain. What are the judge accuracy, the number of questions with at least one correct candidate, and the sensitivity to candidate order in that setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.