acceptodds
Under review as a conference paper at ICLR 2027

Data model refinement for LLM-based SQL code generation

Abstract

The current development of Text-to-SQL systems are mostly based on the improvement of the underlying Large Language Models (LLMs) or the surrounding harness. While this direction is proved to be highly effective for the frontier models, smaller locally deployable solutions usually lack the ability to write suitable complex SQL queries for realistic, complicated databases. This motivated our research to search an alternative direction, where we keep the model intact and optimize the database schema it has to read. The schema becomes the optimization variable, and we search for a data model which is optimal for a small local model to write SQL queries. An evolutionary loop proposes restructured schemas as code, every candidate must preserve the original data, and the fitness of a candidate is the measured execution accuracy of the fixed translator on a training split of the workload and the quality components of the data model such as integrity, implementability, simplicity. On 13 real databases and 101 questions from the Spider 2.0-Lite benchmark, 20 generations of schema evolution raise the mean execution accuracy of three small local translators from 27.1% to 50.6%, at a mutator cost of about $3.50 per database. The two stronger evolved translators were on par with Claude Opus 4.6 with thinking mode reading the original schemas (56.4% and 57.4% against 54.5%). The best schemas are purely additive: they drop nothing, grow a documented layer of derived tables that removes most joins from the translator's job, and differ per model. The whole loop can also run on local hardware: a 27B mutator with thinking mode recovers most of the gains of a frontier mutator at zero API cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Text-to-SQL, schema evolution, database schema design, LLM-driven evolution, data model optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XCVhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper fixes Text-to-SQL translator and optimizes database schema via evolutionary search. It significantly improves the execution accuracy in 13 DBs and 101 questions. But the improvement in held-out is not that much.

Critical strengths

This paper has practical known-workload optimization value. After fixing the translator, it achieves obvious gain and reduces "join" in generated SQL. It provides empirical evidence for reducing task difficulty via derived tables.

Critical weaknesses
  1. It's not very clear about the independent contribution. I think the key gain is mainly from training questions, and lacking the comparison with documentation-only and direct view generation under same supervision and budget. Thus the value of evolutionary search is uncertain.

  2. Row-count check can not guarantee full preservation of source rows/values. Thus, I think it can not support the claimed hard constraint.

Questions for authors

Could you please tell me more about the training/held-out execution accuracy on documentation-only and direct view generation under same training feedback and budget?

1 comment

Thank you for the review and for stating clearly what was missing.

W1 and Question. Independent contribution; documentation-only and direct view generation under the same supervision and budget

To separate these contributions we added an experiment that was not in the submitted version (Section 5.4, Table 2). In it, the schema is written without any question, by the two simpler methods you name. Documentation only (value-format notes that code writes automatically) gives +4.8 points. Direct view generation (one mutator step that sees the database but no question and gets no training feedback) gives +7.5 points. Both together give +10.6 points (95% interval +4.0 to +17.5), for Qwen3.8-27B on 96 benchmark questions.

The evolution with training feedback gives more with the same translator: +26.4 points on the training questions and +14.9 points on 194 unseen questions, which are benchmark questions with one parameter changed and which no mutator saw. So the gain is not limited to the training questions.

Revised paper: Section 5.4; Table 1; Table 2.

W2. Row-count check

We agree, and Reviewer FuYR raised the same point (Q2). We replaced the row count with a full comparison of every source table with the original, in schema and in rows as a multiset in both directions, with a fitness of zero on any difference (Section 3.3). Our answer to Reviewer FuYR, Q2, lists the checks in detail. 39 of the 40 delivered schemas pass. The exception is a database whose own schema declares a foreign key that our rules do not let a candidate declare; its rows are identical (Section 5.3).

Revised paper: Section 3.3; Section 5.3, last paragraph.

We would be glad to answer further questions.

Reviewer_FuYROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for improving Text-to-SQL performance by optimizing the database representation while keeping the SQL-generating language model fixed. Using a mutator LLM to propose schema definitions and migration scripts, with candidate selection driven primarily by execution accuracy on a training workload. The resulting schemas mainly add documentation, derived tables, and views. Experiments cover three local translators and 101 questions from 13 SQLite databases in Spider 2.0-Lite. The main configuration improves accuracy from 27.1% to 50.6% across all questions, while held-out accuracy increases from 24.1% to 28.0%.

Critical strengths
  1. Optimizing database representations for fixed, locally deployed models is a meaningful direction.
  2. The structural analysis provides useful observations about reduced query complexity, and the paper transparently reports limited held-out improvements.
Critical weaknesses
  1. Limited evidence of generalization. It is unclear whether the optimized database helps the model answer new questions or mainly improves performance on the cases used during optimization. In the main experiment, held-out accuracy increases only from 24.1% to 28.0%, much less than the improvement on training questions. With Qwen3.6-27B as the mutator, pooled held-out accuracy even decreases from 24.1% to 22.6%. These results raise concerns that the method mainly produces adaptations tailored to the training workload, with limited benefits for unseen queries.
  2. Data and constraint preservation are not clearly verified. The paper does not sufficiently explain how it checks that the original data and constraints, such as primary keys, foreign keys, and nullability, are preserved after migration. Although static checks are mentioned, their exact rules are unclear. The row-count check alone cannot guarantee that no information is lost.
  3. Pre-join construction and validation need more explanation. It is unclear how the mutator decides which tables to join, which keys and join types to use, and what each row of the resulting table should represent. The paper also does not sufficiently explain how it checks for missing records or duplicated values that could lead to incorrect aggregation.
Questions for authors
  1. What evidence shows that the optimized database helps with new questions beyond those used during optimization? How reliable is the held-out improvement, given that it is measured on only 29 questions?
  2. How do you verify that migration preserves the original data and constraints? Please describe the exact static checks and any post-migration validation, since the row-count check alone cannot guarantee information preservation.
  3. How does the mutator choose which tables to pre-join, the join keys, and the join types? Please provide a concrete example and explain how you check for missing records, duplicated values, and incorrect aggregation in the resulting table.
1 comment

Thank you for the precise questions. The checks of Section 3.3 exist because of them.

Q1. Evidence for new questions; reliability of the held-out improvement

The evidence for new questions is a set of 194 and 187 unseen questions that we added (parameter variations). Each keeps the wording of a benchmark question and changes one parameter, such as a period, a category or a threshold, and no mutator saw them. They gain 14.9 and 15.2 points, which is statistically significant over questions and over databases (Section 5.2).

We agree that 29 questions cannot carry a claim alone, so the gains now have 95% intervals. The 29 held-out questions gain significantly once they are added to the workload and the evolution continues on them: they rise from 46.0% to 72.4% and from 38.1% to 61.9% (Section 5.3).

Revised paper: Section 5.2; Section 5.3; Table 1; Figure 2.

Q2. Preservation of data and constraints

A row count is indeed not enough. The loop now compares every source table with the original after migration: column names and order, declared types, NOT NULL, primary key, foreign keys, unique indexes, and the rows as a multiset in both directions. Any difference gives a fitness of zero. A static check before the build rejects candidates that delete required parts or fill a table from literal rows (Sections 3.2 and 3.3).

39 of the 40 delivered schemas reproduce every source table. The exception is a database whose own schema declares a foreign key to a column without a unique constraint, which our rules do not let a candidate declare; its rows are identical.

Revised paper: Section 3.2; Section 3.3; Section 5.3, last paragraph.

Q3. Pre-joins: tables, keys, join types, row meaning, lost or duplicated records

The mutator chooses the tables, keys and join types from the failed queries in its feedback (Section 3.4). For every added object it must declare the row grain, and optionally the source table whose keys the object covers and the totals it conserves. Code checks each declaration on the data: duplicate keys (a join that multiplies rows), source keys with no row (lost by an inner join or a filter), extra keys, and both sides of every total. A schema with a violation is never delivered.

Appendix B gives the concrete example: a delivered view with its definition and the checker output (22,918 rows, 0 duplicate keys, row count and sum of quantity equal to the source). All 163 added objects of the 25 end schemas declare a grain, and none has a violation (Table 5).

Revised paper: Section 3.3; Section 3.4; Appendix B; Table 5.

Thank you again for the detailed questions.

Reviewer_CHtAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a database-centric approach to improving LLM-based Text-to-SQL generation by optimizing database schemas rather than modifying the underlying language models. The authors introduce an LLM-driven evolutionary framework that iteratively generates and evaluates schema modifications, using SQL execution accuracy as the primary optimization objective. Each candidate consists of a schema definition and a corresponding data migration script, with additional constraints intended to preserve the original data. Feedback from failed SQL executions guides subsequent modifications.

Experiments on 13 databases and 101 questions from Spider 2.0-Lite demonstrate that schema evolution improves the overall execution accuracy of three local translators from 27.1% to 50.6%. The evolved schemas primarily introduce derived tables, views, precomputed aggregations, and additional documentation, substantially reducing the complexity of generated SQL queries.

Critical strengths

S1. Meaningful research direction.

The paper explores a practical alternative to model-centric Text-to-SQL improvements by optimizing database representations for fixed LLMs. This direction is particularly relevant for smaller, locally deployed models and complements existing prompting and fine-tuning approaches.

S2. Coherent execution-guided optimization framework.

The proposed framework integrates schema modification, SQL execution, error analysis, and evolutionary selection into an automated optimization loop. Using actual execution accuracy as the primary feedback signal provides a concrete objective for evaluating schema modifications.

S3. Substantial empirical improvements.

The experiments demonstrate considerable improvements in execution accuracy across three local translators. The analysis of evolved schemas also provides useful insights into how derived tables and precomputed aggregations reduce SQL complexity, particularly the number of joins required.

Critical weaknesses

W1. Insufficient comparison with related schema optimization methods.

The paper lacks comparisons with closely related work, such as The Case for Text-to-SQL Friendly Logical Database Design, which also explores LLM-friendly schema transformations. Since the main contribution appears to be evolutionary search rather than schema optimization itself, comparisons with existing methods are needed to better establish its novelty and effectiveness.

W2. Limited generalization to unseen queries.

Although overall accuracy improves from 27.1% to 50.6%, the gains are largely concentrated on training queries (28.2% to 59.7%), while held-out accuracy improves only from 24.1% to 28.0%. While the authors acknowledge that their approach targets workload-specific optimization, these results raise concerns about its effectiveness on future queries beyond the observed workload. Evaluation across different workload splits would help clarify the generalizability and practical applicability of the evolved schemas.

W3. Unclear sources of performance improvements.

The evolved schemas introduce additional tables, views, and extensive documentation simultaneously. In particular, the substantial increase in schema comments may itself improve the model's understanding of database semantics. Without appropriate ablations, it remains unclear whether the improvements come primarily from schema documentation, structural modifications, or evolutionary search.

W4. Limited evaluation scale and statistical reliability.

The experiments cover only 13 databases and 101 questions, including 29 held-out questions. Although the paper provides statistical tests for SQL complexity reductions, it lacks sufficient statistical evidence for the reported execution accuracy improvements, particularly on held-out queries. Moreover, no repeated evolutionary runs with different random seeds are reported, leaving the stability of the optimization process unclear.

Questions for authors

NA

1 comment

Thank you for the thorough review and for pointing us to the closely related work. We answer the four weaknesses in turn.

W1. Insufficient comparison with related schema optimization methods

We now cite and discuss this paper (Zhang et al.) in Section 2. We share its premise and the use of a workload. Its three transformations are fixed in advance and applied by rule, while ours are open-ended code selected by the measured accuracy of the target translator. Its workload contains past SQL, while ours needs only questions with verified answers. Our layer also contains aggregates, so its correctness has to be checked on the data.

We did not run that method on our databases, and the paper makes no claim about relative accuracy. This is stated in Section 2 and again as a limitation in Section 6. What search adds over a one-off rewrite is partly answered by the experiment without questions (Section 5.4): one question-free rewrite step gives +7.5 points, and the workload-driven evolution gives +26.4 points on the questions it sees and +14.9 on unseen parameter variations (Qwen).

Revised paper: Section 2, paragraph 3; Section 5.4; Section 6.

W2. Limited generalization to unseen queries

Thank you for this point. It led to the main change in the evaluation. In the submitted version the gain was concentrated on the training questions. The revised version shows that the evolved schemas generalize to unseen questions, in two ways.

First, we added 194 and 187 unseen questions of the same workload, the parameter variations. They keep the wording of a benchmark question and change one parameter, and no mutator saw them. Their accuracy rises from 43.5% to 58.4% (Qwen3.8-27B) and from 35.1% to 50.3% (Gemma-4-31B-it), with intervals that exclude zero over questions and over databases.

Second, a new stage continues the evolution when new questions arrive. The held-out benchmark questions then rise from 46.0% to 72.4% and from 38.1% to 61.9%, and the earlier questions keep their level.

The scope of this generalization is the observed workload: held-out benchmark questions that ask for a different analysis change by +6.9 and +1.2 points before the second stage, with intervals that include zero. Three rotating splits of one database give the same picture (Appendix D).

Revised paper: Section 4.3; Section 5.2; Section 5.3; Table 1; Figure 2; Appendix D.

W3. Unclear sources of performance improvements

To answer this we added an experiment that was not in the submitted version (Section 5.4, Table 2 now reports). In it, no question is shown to anything that writes the schema, so it measures what documentation and structure give without the workload. Automatic documentation of value formats alone gives +4.8 points. One question-free restructuring step gives +7.5 points. Both together give +10.6 points, which is significant over questions and over databases.

This experiment was run with one translator (Qwen3.8-27B). Its question set differs from that of the evolution, so the numbers do not split the gain of the evolution into parts.

Revised paper: Section 5.4; Table 2; Figure 2; Appendix D.

W4. Limited evaluation scale and statistical reliability

The submitted version had no statistics. The accuracy gains in Table 1, Table 2 and Figure 2 now have 95% paired bootstrap intervals over questions (Section 4.3). The gains that the paper claims are statistically significant: +26.4 and +36.8 points on the training questions, +14.9 and +15.2 points on the unseen parameter variations, and +26.4 and +23.8 points on the held-out questions after the second stage. The gain on the parameter variations is also significant when whole databases are resampled and under a two-sided exact sign test (Section 5.2).

The benchmark subset is the same, 13 databases and 101 questions, and we added 194 and 187 parameter variations. We could not repeat the evolution with different seeds in the discussion period; Section 6 states this.

Revised paper: Section 4.3; Table 1; Figure 2; Section 6.

We would be glad to answer further questions on any of the four points.

Comments on the paper

1 comment

We sincerely thank the three reviewers for their careful reviews. Based on these comments, we could improve results significantly and address the limitations which we were already mentioned in the previous version.

The reviews raised four main points: the gain was concentrated on the training questions, data preservation was not verified strongly enough, the correctness of pre-joined tables was not checked, and the accuracy numbers had no statistics. We agree with all four and revised the paper on each of them.

We are especially grateful to Reviewers FuYR and XCVh for the comments on the row-count check. Following them, we replaced the row count with a full comparison of every source table and added checks for the derived tables. Since these checks decide which schemas are valid, we ran the experiments again with the revised loop, and Section 5 reports these results. The idea of the paper, the benchmark, the databases and the questions are unchanged. A marked diff against the submitted version is uploaded with the revision.

The revised experiments cover fewer configurations than the submitted ones, for practical reasons. With our compute we could not repeat every configuration under the new checks within the discussion period, so we reduced the set to two translators (Qwen3.8-27B and Gemma-4-31B-it) and one mutator. The third translator, the frontier reference translators and the local mutators are not part of the revision. The mutator is now GPT 6 Astra instead of Claude Sonnet 4.6, because this was the frontier model we had access to during the revision.

What is new in the method (Section 3):

  • Source tables are frozen. After migration, code compares every source table with the original: columns, declared types, NOT NULL, primary key, foreign keys, unique indexes, and the rows as a multiset in both directions. Any difference gives a fitness of zero (Section 3.3).
  • Stored answers are rejected in code (Section 3.3), and comments that instruct the translator are deleted by pattern (Section 3.4).
  • Every added table or view declares its row grain, and optionally the source keys it covers and the totals it conserves. Code checks duplicate keys, empty keys, lost keys, extra keys and both sides of every total (Section 3.3; Appendix B has a worked example).
  • The loop trains on families of variations of the training questions, selects the delivered schema with a validation set, and prefers the shorter schema (Section 3.5).
  • A second stage continues the evolution when new questions arrive (Section 3.6).

What the revised experiments show (Section 5; 13 databases, 101 benchmark questions, two translators):

  1. Known workload. Training questions rise from 52.3% to 78.7% for Qwen3.8-27B (+26.4 points, 95% interval +14.8 to +38.0) and from 43.1% to 79.9% for Gemma-4-31B-it (+36.8, interval +24.5 to +49.0).
  2. Held-out benchmark questions do not gain before the evolution has seen them: +6.9 points (interval -8.0 to +23.0) and +1.2 (-10.7 to +14.3). The abstract and Section 5.2 now say this.
  3. Unseen parameter variations of the benchmark questions gain 15 points with both translators. These questions keep the wording of a benchmark question and change one parameter, and no mutator saw them. Qwen: 43.5% to 58.4% on 194 questions (+14.9, interval +8.6 to +21.5; over databases +3.3 to +26.7). Gemma: 35.1% to 50.3% on 187 questions (+15.2, interval +9.1 to +21.4; over databases +4.6 to +30.4). The gain is on variations of training questions (+16.0 and +18.9). Variations of held-out questions do not gain, so the paper claims generalization inside the observed workload only.
  4. Continued evolution. When the held-out questions are added, they rise from 46.0% to 72.4% (Qwen) and from 38.1% to 61.9% (Gemma), and the training questions keep their level (82.4% and 78.9%). The second stage was accepted in 8 of 13 and 7 of 12 runs; in the other runs the schema stays as it was.
  5. Schemas written without questions (Qwen, 96 questions, two evaluations). Automatic value-format notes: +4.8 points (interval -0.2 to +10.2). One mutator step that sees no question: +7.5 (+1.0 to +14.6). Both: +10.6 (+4.0 to +17.5; over databases +2.0 to +20.5).
  6. Checks. 24 of 25 stage-1 schemas and all 15 accepted stage-2 schemas reproduce every source table, and no delivered schema has a contract violation. The one exception is a database whose own schema declares a foreign key that our rules forbid; its rows are identical.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.