- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Type-Constrained Tokenization (TCT), which compiles database schemas into scope-aware menus of table and column choices and converts SQL into action sequences for model training. The system automatically supplies forced actions, filters inadmissible choices during generation, and renders the output as SQL. The authors primarily compare plain SQL, menu-index text, and compiled actions on Spider, with additional ablations, perturbation tests, and candidate-selection experiments. TCT substantially reduces execution failures, but an independent accuracy advantage from its tokenization has not been established. Candidates generated using different representations also exhibit some complementarity.
Critical strengths
-
Clear improvement in execution reliability. On the common set of 1,863 test questions, TCT produces 6 execution failures, compared with 324 for plain SQL. Even when unsupported questions are counted as failures, the full-test-set failure count decreases from 405 to 245. These results demonstrate that the system effectively reduces invalid queries.
-
The representation accommodates unseen database schemas. TCT represents table and column references using reusable menu indices. New databases can use the same model by rebinding the menus, without adding dedicated tokens for every new table or column name. Evaluation on 40 unseen databases provides a concrete demonstration of integrating database constraints with a pretrained model.
Critical weaknesses
-
The independent contribution of the specialized tokenizer is not sufficiently isolated. The paper lacks a direct comparison against menu-index text with constraints equivalent to TCT’s. The mask ablation demonstrates the importance of inference-time filtering, but does not establish the additional benefit of specialized tokenization over constrained text representations. The earlier constrained-decoding experiment in §5.5 is also not a matched comparison, leaving the central contribution insufficiently isolated.
-
Training data and checkpoint-selection budgets are not fully matched. Plain SQL uses 6,106 second-stage training examples, whereas TCT and menu-index text use 5,335. TCT selects the best development checkpoint among five rehearsal seeds, while the plain-SQL anchor is fixed beforehand. Matching optimization steps does not eliminate these differences, making it difficult to attribute accuracy changes solely to the output representation.
-
The absence of an execution-feedback repair baseline leaves TCT’s practical value uncertain. SQL agents (e.g., MAC-SQL) can already repair queries using database error messages. It therefore remains unclear whether TCT offers substantial performance gains within current agent-based workflows.
-
Grammar coverage and evaluation scope limit the applicability of the findings. The implementation supports constructs such as filtering, joins, aggregation, and uncorrelated subqueries, but excludes self-joins, set operations, and correlated subqueries. The main three-model comparison covers only 1,863 of 2,147 test questions, and the evaluation primarily uses Spider and its perturbation suites. Evidence for more complex SQL and other data distributions therefore remains limited.
-
Insufficient detail about training-data construction limits reproducibility. The paper uses schema-literacy, Spider, and rehearsal stages, but does not sufficiently describe the data sources, construction templates, sample counts, or cross-stage overlap for the first and third stages. The description is insufficient to reconstruct the full training pipeline, and the contribution of each stage is not independently analyzed (§4, Appendix B).
Questions for authors
-
Does the specialized tokenizer provide benefits beyond constraint filtering? Menu-index text could also be decoded under equivalent grammar and schema constraints. Could you compare this constrained text baseline with TCT under matched conditions? This comparison would help determine whether TCT’s gains come from its specialized tokenization or from inference-time constraints.
-
Do the main findings hold under matched training and selection conditions? Could you provide comparisons using identical training examples and development-set selection budgets, together with independent repetitions of the full training pipeline? The existing rehearsal-seed controls do not capture variability across the entire training process.
-
What advantages remain over execution-feedback repair? How do plain SQL and menu-index text perform after receiving specific database error messages and being allowed one or two repair rounds? Please also report generation volume, database calls, or measured latency to assess the performance–cost tradeoff.
-
How would the system handle unsupported queries in deployment? Without access to reference SQL, how does the system handle requests beyond its grammar? Could you evaluate whether the fallback or judge gains persist on the full test set using a routing procedure that does not rely on reference answers?
-
Could you provide the complete data-construction procedure? Please specify the data sources, sample counts, construction and filtering rules, and database- and example-level overlap across the three stages. A complete example showing how the same input is converted into all three training targets would also be helpful.
-
How robust are the accuracy findings under a consistent scoring protocol? Building on the reported sensitivity analyses, could you summarize results on the same test set using scoring that respects ORDER BY, and provide database-clustered confidence intervals for the main accuracy differences? This would clarify which comparisons support accuracy gains and which primarily support reductions in execution failures.