acceptodds
Under review as a conference paper at ICLR 2027

SQaLe: A Large Realistic Dataset to Empower Small Specialized Text-to-SQL Models

Abstract

The strong coding generation capabilities of large language models have accelerated progress in text-to-SQL, the task of converting natural language questions into executable SQL queries. The state-of-the-art in text-to-SQL is currently set by frontier language models embedded in composite agentic pipelines that incur substantial inference cost. Specialised small models would avoid this cost but training them requires data reflecting the scale, semantics, and structural diversity of real-world databases. Towards this vision, we introduce SQALE, a large-scale semi-synthetic text-to-SQL dataset built on 9,259 schemas from SchemaPile, a collection of real-world schemas extracted from GitHub. We present an informed data generation pipeline that combines schema extension, value synthesis, question generation, and agentic answering under execution-based validation at every stage, and produce 1,408,056 high-quality natural language (NL) questions pair with 176,761 corresponding SQL queries. Besides the large scale of SQaLe, our analysis shows that it offers significantly larger schemas than existing datasets and rich query distribution. We demonstrate how SQaLe empowers small specialised text-to-SQL models by training a small language model with GRPO directly on SQaLe that improves the untrained base model by nearly 33 points on BIRD dev, which is an unprecedented gain. We find that, among other factors, key to this performance is that training on SQaLe enables the model to learn exploring the database. It finds the relevant tables more often and looks up far more of the values it needs than when trained on other datasets. The dataset, generation pipeline, and training code are available at: url_disclosed_upon_acceptance.

Primary area:
datasets and benchmarks
Keywords:
text-to-SQL, Dataset, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pjfPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduced SQALE, a large-scale semi-synthetic text-to-SQL dataset (1.4M questions + 176k queries) built on 9k real-world schemas extracted from GitHub. Defined the critieria used to design the generation pipline, including schema expansion, value synthesis, question generation, etc. Followed by some analysis around the dataset, e.g., testing results with model trained on different datasets. The results shows SLM trained on SQaLe achieves good performance.

Critical strengths

Strengths

S1. The paper introduces a large-scale text-to-SQL dataset, including 1.4M questions and 176k SQL queries, based on seed schemas obtained from real-world problems.

S2. Well-defined design criteria and structured generation pipeline. The paper listed 12 dataset design criteria covering schema-level, data-level, and query-level properties. This definition improves the clarity.

Critical weaknesses

W1. Limited quality guaruntee. Only one human validation was done in the paper, judge verdicts against human labels, on 225 judge calls by one author. Moreover, it's done on BIRD-dev not SQaLe, leaving a potential distribution gap. The reliability of the dataset is then questioned, especially for complex queries, and underspecified paraphrases (ambiguous).

W2. The performance improvements might be biased and not significant on BIRD. I'm not convinced whether the improvement can be attributed specifically to the dataset improvement, rather than the test distribution simlarity, training curriculum, etc. Other tests suggests that the gains on SQaLe may not translate uniformly to external benchmarks. MSQaLeM_{SQaLe} and MBIRDM_{BIRD} achieve relative similar performance on both BIRD and EHRSQL, considering SQaLe has way more training samples and BIRD-test is relatively simple.

Questions for authors

Q1. How does the training curriculum affect the model training with SQaLe, does it require careful design and a bunch of testing? Is this GRPO training stable?

Q2. How's the model performance on complex queries with human annotated dataset, like Spider 2.0

Q3. Has the author trained models with data from different difficulty level, and how does the complexity affects the performance across levels.

Q4. What's the cost of data creation, and model training cost?

Q5. Can the authors share some examples of the ambiguous queries, and what is expected to handle these queries.

4 comments
AuthorsOpenReview ↗

Response Part 1/2

Thank you for the constructive feedback and questions, which we gladly address below.

W1 – Quality validation

We consider BIRD-Dev a strong proxy for assessing the reliability of the LLM judge as it provides us with a relatively large set of human-generated SQL queries. We further verify verify the quality of the dataset with a strong LLM judge from another model family (Claude Sonnet 5.5) which marks 1 of 75 SQaLe question–SQL pairs as incorrect (none of the 25 hard cases), against 8% of human-written SQL in the BIRD dev gold set.

Although we envision the training of small text-to-SQL models as the main use-case of the SQaLe dataset and not benchmarking them, these assessments confirm that SQaLe provides a fruitful basis for a new benchmark. Importantly, to validate the quality of SQaLe for its intended use of model training, we experimentally demonstrate that a model trained on SQaLe generalizes well to the key benchmarks in the field.

For short-and-ambiguous rewrites in SQaLe, the independent LLM judge judges the reference SQL most plausible in 24 out of 25 cases. We further assess the quality utility by downstream evaluation on an ambiguity benchmark, AMBROSIA [1]. The SQaLe-trained model returns a valid interpretation of an ambiguous question for 48.3% of 300 questions, against 33.7% for the BIRD-trained model (+14.7 pp), without losing accuracy relative to the unambiguous rewrites.

W2 – Model performance evaluation

The SQaLe-trained model does indeed not outperform the BIRD-trained model on BIRD dev (52.3 vs 54.7). While the BIRD-trained model is of course in-domain and the SQaLe-model generalizes relatively well out-domain, our manual analysis shows that the gap is mainly driven by answer formatting rather than incorrect answering. That is, on 37 of the 199 BIRD dev questions that only the BIRD-trained model gets right, the SQaLe-trained model returns exactly the gold rows but adds extra columns (for example the count it ranked by), which BIRD's metric scores as incorrect. This happens 79 times for the SQaLe-trained model against 24 times for the BIRD-trained one.

If we would score the models accordingly, the two are on-par (55.4% vs 55.5%). This is a notable result, as SQaLe is not trained on the BIRD-Train corpus and is therefore out-of-domain, which demonstrates the generalization power of SQaLe. We keep BIRD's official metric but will add this reflection. Moreover, SQaLe's generalization performance shows across evaluation datasets: on unseen SQaLe schemas (+13.8 pp), Spider 1.0 dev (+13.8 pp), WikiSQL (+9.7 pp), EHRSQL (+11.4 pp).

Questions

Q1 Influence training curriculum The curriculum is a fixed schedule, not tuned per run (Appendix H): questions are bucketed by gold-SQL length (50th/80th/95th percentiles), the first 400 steps draw only short questions, and the mixture then anneals linearly towards the long tail over 1,000 steps, with a floor of 0.15 per bucket so short queries keep contributing gradient. In earlier runs, this notably improved performance on hard and long queries, whose signal is otherwise averaged away by the many short ones (half of SQaLe's gold queries are under 150 characters). Starting close to the natural mixture matters for GRPO, since groups in which all candidates fail carry no advantage signal.

After initial instabilities, training was stabilized by (Appendix I): token-level PPO clipping with DAPO clip-higher (ε = 0.2 / 0.28) and an importance-ratio cap of 2.0; a k3 KL penalty adapted to a target of 0.005, with the reference re-anchored every 111 updates; Dr. GRPO-style advantages clipped to ±2 with negatives weighted 0.7; DAPO resampling of zero-variance groups; entropy-triggered guards on the KL weight and learning rate; and on-policy updates at temperature 1.0 with gradient clipping, bounded rewards and episode caps. All three models use these identical hyperparameters, apart from the curriculum strength.

Q2 Complex queries as in Spider2

We evaluated all models on the part of Spider 2.0-lite that runs on local SQLite files (75 of 547 instances; the rest need BigQuery, Snowflake or DuckDB), with three decoding seeds and Spider 2.0's own scorer. All 2B models are at the floor: SQaLe-trained 5.8%, BIRD-trained 4.9%, SynSQL-trained 2.2%, base 1.3%. These enterprise tasks are beyond a 2B agent in this protocol for every training corpus, which we will discuss as future work.

On the human-annotated Spider 1.0 dev set, the SQaLe-trained model is clearly the most accurate (76.8% vs 63.1% for the BIRD-trained model, +13.8 pp, 298 questions), also under perturbation: Spider-Syn 69.1% vs 59.4%, Spider-Realistic 71.8% vs 62.1%.

AuthorsOpenReview ↗

Response Part 2/2

Q3 Influence difficulty levels We trained on all difficulty levels jointly, without per-level ablations; the length curriculum (Q1) shifts training towards hard questions over time, as the two long buckets are mostly labelled hard (59% and 76%). On 420 unseen SQaLe test questions (140 per level), the SQaLe-trained model scores 69.3 / 48.6 / 30.7% on simple / moderate / hard (base model: 40.0 / 20.0 / 12.1%). Its advantage over the other trained models is largest at moderate (+23.6 pp) and still +11.8 pp at hard. We will add this breakdown to the paper.

Q4 Cost of data generation and training In H100 GPU-hours (GPUs allocated × wall-clock time):

  • Data creation: all LLM calls ran on self-hosted open-weight Qwen models (Appendix M), so there were no API costs. The 241 generation shards (2 GPUs each) took 1,432 GPU-hours, including ~10% on tasks that timed out or crashed.
  • Training: one GRPO run of the 2B model (3 × H100 for 50 h) costs 150 GPU-hours, identical for the SQaLe-, BIRD- and SynSQL-trained models.

We will report these budgets in the paper in the appendix.

Q5 Ambiguous query style examples

  • Original: "Find the queue names and their maximum job limits for all queues associated with compute resources that have a maximum memory per node greater than 256 GB."
  • Short and ambiguous: "Need queue names and job limits for the heavy-memory compute setups."
  • Reference SQL: SELECT bq.QUEUE_NAME, bq.MAX_JOB_IN_QUEUE FROM BATCH_QUEUE bq JOIN COMPUTE_RESOURCE cr ON bq.COMPUTE_RESOURCE_ID = cr.RESOURCE_ID WHERE cr.MAX_MEMORY_NODE > 256;

Here, "heavy-memory" leaves the 256 GB threshold implicit.

With the schema withheld, the agent is expected to resolve such underspecification by exploring tables, values and join paths, and to answer on a best-effort basis. Keeping a single reference SQL is a deliberate scoping decision, as our setup has no clarification channel.

Beyond best-effort answering, we expect these questions to be used for building disambiguation pipelines. Every ambiguous rewrite is labelled with its style and paired with its fully specified original and the shared reference SQL, so the difference between the two questions is exactly the information a clarification step has to recover (here, the 256 GB threshold). This provides supervision for detecting underspecified requests, asking the right clarification question, and mapping the clarified request to SQL; users can also select or exclude the style. The style is already useful in its current form: on the ambiguity benchmark AMBROSIA [1], the SQaLe-trained model returns a valid interpretation for 48.3% of 300 ambiguous questions, against 33.7% for the BIRD-trained and 27.0% for the SynSQL-trained model. We will add examples of all seven styles to the appendix.

[1] Saparina, Irina, and Mirella Lapata. "Ambrosia: A benchmark for parsing ambiguous questions into database queries." Advances in Neural Information Processing Systems 37 (2024): 90600-90628.


We hope that we have addressed your remarks and questions and that our response positively contributes to your assessment: we are very excited about the SQaLe dataset and how it materializes the potential for training small specialized text-to-SQL models. We also welcome any remaining questions you may have.

Thank you for your response. The testing or human evaluation of LLM judges can not serve as a definitive ground truth validation of the dataset, and sample size are not enough to support the claim. I'll maintain my score.

Thank you for the follow-up. We agree that validating the judge is not a definitive check for the ground-truth accuracy of the dataset and we do not claim that SQaLe is error-free. What we show in the paper is that SQaLe is a reliable training corpus and a better basis for training small text-to-SQL models than prior datasets. Like other training corpora such as FineWeb [1], we establish this through controlled training comparisons. All models are trained from the same base model and use the same number of training steps, batch size and hyperparameters, apart from the curriculum schedule. To that end, the full evaluation results on external benchmarks show that MSQaLeM_{SQaLe} performs best in 9 of 10 evaluations. The only exception to this is the BIRD dev dataset, where MBIRDM_{BIRD} has an in-distribution advantage from training on BIRD’s own training split.

Full results:

BenchmarkQuestionsMSQaLeM_{SQaLe}MBIRDM_{BIRD}MSynSQLM_{SynSQL}Base (Qwen3.5-2B)
Unseen SQaLe test schemas42049.535.729.524.0
BIRD dev (full)1,53250.153.941.428.3
Spider 1.0 dev29876.863.164.440.9
Spider-Syn29869.159.460.438.3
Spider-Realistic29871.862.161.143.3
WikiSQL30066.757.037.026.7
EHRSQL (all)30025.321.015.07.0
EHRSQL (gold not a single NULL/0 row)21030.519.012.95.7
AMBROSIA, ambiguous questions (any valid reading)30048.333.727.013.0
Spider 2.0-lite, local SQLite subset (mean of 3 seeds)755.84.92.21.3

As described above, we also

  1. build quality control into the dataset generation as every pair must execute against the database and be accepted by an LLM judge that we validated against human labels.
  2. perform independent post-hoc checks of SQaLe with a judge from a different model family that rates the instances in SQaLe as at least as accurate as human-annotated instances in the BIRD dataset. We consider this a more informative analysis than comparing against human labels because human-written gold SQL is not a clean reference in complex text-to-SQL settings. [2] find substantial annotation errors of 52.8% in BIRD dev and 66.1% in Spider 2.0-Snow, a result that is in line with per-domain findings by [3]. These error analyses consistently show that a large share of supposed model failures turn out to be correct SQL scored against wrong or overly strict gold labels [4, 2], which is in line with our manual evaluation of our judge that correctly disagreed with BIRD’s gold SQL in 52/83 cases (Sec 2.3). LLM-based reviewers have also been shown to surface annotation errors that human annotation misses and proposed as effective tools in creating robust datasets [2].

[1] Penedo et al. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. NeurIPS 2024 Datasets and Benchmarks Track.

[2] T. Jin, Y. Choi, Y. Zhu, D. Kang. Text-to-SQL Benchmarks are Broken: An In-Depth Analysis of Annotation Errors. CIDR 2026. (Extended version: Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards, arXiv:2601.08778)

[3] N. Wretblad, F. Riseby, R. Biswas, A. Ahmadi, O. Holmström. Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark. ACL 2024.

[4] M. Pourreza, D. Rafiei. Evaluating Cross-Domain Text-to-SQL Models and Benchmarks. EMNLP 2023.

Reviewer_ku1DOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SQaLe, a large-scale semi-synthetic text-to-SQL dataset designed to support the training of small, specialized language models. Built upon real-world database schemas from SchemaPile, SQaLe contains approximately 1.4 million natural language questions associated with 176,761 distinct SQL queries across 9,259 schemas. The authors propose a multi-stage data generation pipeline involving schema extension, synthetic table population, question generation, agentic SQL generation, execution-based validation, LLM-based judging, and question reformulation. To demonstrate the dataset's utility, the authors train Qwen3.5-2B directly from its base checkpoint using GRPO and compare models trained on SQaLe, BIRD, and SynSQL under largely shared training configurations. The SQaLe-trained model achieves 66.3% execution accuracy on the SQaLe test set and 52.3% on BIRD dev, improving substantially over the untrained base model. Further analyses suggest that training on SQaLe improves the model's ability to discover relevant tables in large schemas and retrieve database values needed for SQL generation. The paper argues that realistic schema complexity, populated databases, and domain diversity are important for developing effective small text-to-SQL agents.

Critical strengths
  1. Large and diverse text-to-SQL dataset. The authors build SQaLe using 9,259 real-world database schemas, with over 1.4 million natural language questions and 176,761 SQL queries. Compared with existing datasets, SQaLe contains larger schemas, with a median of 113 tables per database. The authors also design a complete data generation process that includes schema extension, table value generation, question generation, and SQL validation. This work provides useful training data for small text-to-SQL models.

  2. Strong results and useful analysis of database exploration. The authors train a 2B model with GRPO and achieve a 33.0-point improvement on BIRD dev over the base model. The SQaLe-trained model also achieves 66.3% accuracy on the SQaLe test set, compared with 54.0% and 50.7% for models trained on BIRD and SynSQL. The authors further study how models find relevant tables and search for values in databases. These results provide useful evidence that training on SQaLe can improve both SQL generation and database exploration.

Critical weaknesses
  1. Limited validation of the dataset's semantic correctness. The authors mainly use an LLM judge to check whether the generated SQL queries correctly answer the questions. They test the judge on 225 calls from BIRD dev, but manually review only 83 cases where the judge and execution accuracy disagree. The SQL generator and judge also use the same model, which may lead to similar errors. More importantly, the authors generate seven new versions of each question and keep the original SQL without checking whether the meaning stays the same, especially for ambiguous questions. As a result, the current experiments do not fully show the quality of the final SQaLe dataset. The authors should provide an independent human evaluation of SQaLe samples, including the rewritten questions.

  2. Limited evidence for generalization and the sources of performance gains. Although SQaLe performs well on its own test set, its results on other benchmarks are less clear. In Table 4, the SQaLe-trained model achieves 52.3% accuracy on BIRD dev, below the 54.7% of the BIRD-trained model, while both models achieve 23.7% on EHRSQL. In addition, SQaLe differs from the other datasets in several ways, including schema size, table values, domain coverage, and question types. The training methods also use different curriculum settings. Without further experiments that study these factors separately, it is difficult to tell which factors lead to the improvements in database exploration and SQL generation.

Questions for authors

Q1. The authors test the SQL judge on 225 calls from BIRD dev, but manually review only 83 cases where the judge and execution accuracy disagree. Have the authors also checked question-SQL pairs directly sampled from SQaLe? Could they provide human evaluation results for different SQL difficulty levels and query types to better support the quality of the dataset? Q2. The authors rewrite each original question into seven different styles while keeping the same SQL. How do the authors ensure that these rewritten questions still match the original SQL? In particular, the "short and ambiguous" style may remove important conditions from the question. Could the authors provide results that show how often these rewritten questions keep the original meaning? Q3. SQaLe differs from BIRD and SynSQL in schema size, table values, domain coverage, and question types. The authors also use different curriculum settings for the three datasets. Could the authors provide further experiments to show which factors contribute most to the improvements? In particular, how much do larger schemas and richer table values help when other factors remain the same? Q4. In Table 4, the SQaLe-trained model achieves 52.3% accuracy on BIRD dev, below the 54.7% of the BIRD-trained model, while both models achieve 23.7% on EHRSQL. These results do not clearly show that SQaLe provides better results on external benchmarks. How do the authors explain these results, and could they provide more evidence of generalization to unseen databases and domains? Q5. The authors compare SQaLe with Spider 2.0 in terms of dataset statistics, but do not evaluate the trained models on Spider benchmarks. Since Spider 2.0 includes complex real-world database tasks, why did the authors not include it in the evaluation? Could they provide results on Spider 1.0 or a suitable subset of Spider 2.0 to further test the model's ability to handle complex SQL tasks? Q6. The authors split SQaLe into training and test sets at the schema level. Since SchemaPile contains schemas from GitHub repositories, could different schemas from the same repository appear in both sets? Have the authors checked for duplicate or highly similar schemas across the two sets? This would help clarify whether the test results reflect the model's ability to work with unseen database structures.

2 comments
AuthorsOpenReview ↗

Response Part 1/2

Thank you for the constructive feedback and questions, which we gladly address below.

W1 & Q1 – Quality validation

We consider BIRD-Dev a strong proxy for assessing the reliability of the LLM judge as it provides us with a relatively large set of human-generated SQL queries. We further verify verify the quality of the dataset with a strong LLM judge from another model family (Claude Sonnet 5.5) which marks 1 of 75 SQaLe question–SQL pairs as incorrect (none of the 25 hard cases), against 8% of human-written SQL in the BIRD dev gold set. For short-and-ambiguous rewrites in SQaLe, the independent LLM judge judges the reference SQL most plausible in 24 out of 25 cases.

Although we envision the training of small text-to-SQL models as the main use-case of the SQaLe dataset and not benchmarking them, these assessments confirm that SQaLe provides a fruitful basis for a new benchmark. Importantly, to validate the quality of SQaLe for its intended use of model training, we experimentally demonstrate that a model trained on SQaLe generalizes well to the key benchmarks in the field.

W2 & Q4 – Generalization

The SQaLe-trained model does indeed not outperform the BIRD-trained model on BIRD dev (52.3 vs 54.7). While the BIRD-trained model is of course in-domain and the SQaLe-model generalizes relatively well out-domain, our manual analysis shows that the gap is mainly driven by answer formatting rather than incorrect answering. That is, on 37 of the 199 BIRD dev questions that only the BIRD-trained model gets right, the SQaLe-trained model returns exactly the gold rows but adds extra columns (for example the count it ranked by), which BIRD's metric scores as wrong. This happens 79 times for the SQaLe-trained model against 24 times for the BIRD-trained one.

If we would score the models accordingly, the two are on-par (55.4% vs 55.5%). This is a notable result, as SQaLe is not trained on the BIRD-Train corpus and is therefore out-of-domain, which demonstrates the generalization power of SQaLe. We keep BIRD's official metric but will add this reflection. Moreover, SQaLe's generalization performance shows across evaluation datasets: on unseen SQaLe schemas (+13.8 pp), Spider 1.0 dev (+13.8 pp), WikiSQL (+9.7 pp), EHRSQL (+11.4 pp).

Q2 – Rephrasing consistency

The "short and ambiguous" style is intentionally underspecified to support robustness training and the development of downstream disambiguation pipelines. We validate the intent alignment as follows.

  • With the same LLM judge, the fully specified queries add 0–4.6 pp of flagged errors over the original question, and the intentionally underspecified styles add 8.1 (short high-level) and 13.6 pp (short and ambiguous).
  • We also measured fidelity: on 90 unseen test questions asked in all eight phrasings, whenever a model can solve the original, some model reaches the original's gold result from the rewrite alone: in 64–92% of cases for the six fully specified or paraphrased styles, and in 34% for the short-and-ambiguous style whose underspecification is intentional.
  • The intentionally ambiguous short-and-ambiguous style has a measurable benefit: on the external AMBROSIA benchmark [1] the SQaLe-trained model returns a valid interpretation of ambiguous questions far more often than the BIRD-trained one (48.3% vs 33.7%).

Q3 – Dataset ablations

We agree that controlled ablations on the SQaLe-trained model would be informative to surface the value of the different dataset properties and training curricula, but this would be very compute intensive. Instead, we inform the design desiderata by prior research and instill and analyze these properties in the dataset (Sec 2.1, 2.2, and 3). While we are reserved in specific attributions to dataset properties beyond schema sizes, table content and coverage (as surfaced in Section 4.3), the performance and generalization performance of the SQaLe-trained model versus SynSQL and BIRD-train demonstrate the value of the informed design of SQaLe well.

Q5 – Dataset comparison

We evaluated all models on the part of Spider 2.0-lite that runs on local SQLite files (75 of 547 instances; the rest need BigQuery, Snowflake or DuckDB), with three decoding seeds and Spider 2.0's own scorer:

  • All 2B models are at the floor there: SQaLe-trained 5.8%, BIRD-trained 4.9%, SynSQL-trained 2.2%. These heavy enterprise tasks are beyond a 2B agent in this protocol, for every training corpus, which we will elaborate on in the future work section.
  • Spider 1.0 dev, whose small human-written schemas are the regime SQaLe does not explicitly target: the SQaLe-trained model is clearly the most accurate, 76.8% vs 63.1% for the BIRD-trained model (+13.8 pp) on 298 questions. It is also most accurate when questions are perturbed: 1) with synonyms (Spider-Syn): 69.1% vs 59.4%, 2) with explicit column mentions removed (Spider-Realistic): 71.8% vs 62.1%.
AuthorsOpenReview ↗

Response Part 2/2

Q6 – Data splits

Thank you for suggesting to check for any influence of schema-level splitting rather than repository-level: Our dataset split is at schema level, so none of the test schemas is in the train split. We checked the overlap in source repositories: 197 schemas in the test-set share a source repository with a train-split schema. Excluding these schemas from the test set does not reduce the SQaLe-trained model's performance on the test split, so there is no repository-derived domain leakage.

Moreover, we validated that no test schema has a near-duplicate in the train split (table-name Jaccard ≥ 0.8: 0 of 423; identical DDL: 0). Test schemas are also not closer to train than train schemas are to each other, and even test schemas from a shared repository are structurally different from their train neighbours (median Jaccard 0.02).


[1] Saparina, Irina, and Mirella Lapata. "Ambrosia: A benchmark for parsing ambiguous questions into database queries." Advances in Neural Information Processing Systems 37 (2024): 90600-90628.

We hope that we have addressed your remarks and questions and that our response positively contributes to your assessment: we are very excited about the SQaLe dataset and the vision it enables. We also welcome any remaining questions you may have.

Reviewer_b2sYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SQaLe, a semi-synthetic training dataset for small, specialized Text-to-SQL models. Its construction pipeline starts with real database schemas sourced from GitHub through SchemaPile, then applies schema expansion, table data synthesis, question generation, tool-assisted SQL generation, execution and semantic filtering, and question-style rewriting to produce the final corpus.

The dataset contains 9,259 database schemas, 176,761 SQL queries, and 1,408,056 question–SQL pairs. A distinguishing feature is its large schema size: the median schema contains 113 tables and 538 columns. The databases are also populated with synthetic data, allowing models to explore table structures and specific values through tools.

Starting from the Qwen3.5-2B base model, the authors perform GRPO training separately on SQaLe, SynSQL, and BIRD. On the 300 evaluated BIRD dev questions, the SQaLe-trained model improves execution accuracy from the base model’s 19.3% to 52.3%; it achieves 66.3% on the evaluated SQaLe test examples. Further analyses examine table discovery, value lookup, and performance across schema sizes, suggesting that large schemas, richer table contents, and domain diversity help models learn database exploration behaviors.

Critical strengths
  1. The dataset addresses an important and specific gap in existing training resources for small models.

SQaLe provides not only additional question–SQL pairs but also large relational schemas and queryable database contents, requiring models to discover relevant tables, inspect join relationships, and look up specific values. This is closer to applications requiring database exploration than training models to generate SQL directly from a fully provided DDL.

The data synthesis pipeline also incorporates dependency ordering, cyclic foreign-key handling, structural checks, repairs, and non-uniform foreign-key distributions. Although some realism claims still require validation, the resource and its construction pipeline have potential reuse value.

  1. The small-model training experiments provide meaningful evidence of cross-dataset improvements.

On the evaluated BIRD sample, the SQaLe-trained 2B model achieves 52.3%, substantially improving over the base model’s 19.3% and outperforming the SynSQL-trained model’s 44.3%. On EHRSQL, it achieves 23.7%, compared with 8.2% for the base model and 13.3% for the SynSQL-trained model.

These results support SQaLe’s value as a training resource that improves model performance, beyond simply being a large generated corpus. The authors also compare table discovery and value lookup behaviors, providing useful clues about the sources of improvement.

Critical weaknesses
  1. Validation of the claimed high quality of the full SQaLe corpus is insufficient, particularly regarding semantic consistency after question rewriting.

The 94.5% acceptance accuracy reported in Section 2.3 comes from 225 judge calls generated from 158 BIRD questions, rather than a random sample from SQaLe. Cases where execution results and judge decisions disagreed were further examined by one author. This demonstrates some practical usefulness of the judge, but it does not directly estimate the correctness of the final SQaLe corpus.

This limitation is particularly important for the seven question-rewriting styles. Rewritten questions inherit the original SQL, while the “short and ambiguous” style explicitly omits some specifications. Such a question may reasonably admit multiple query interpretations, meaning that the original SQL may no longer be the uniquely appropriate supervision target. The paper does not sufficiently explain how rewrites are checked for changes to conditions, aggregation scope, or answer granularity.

Moreover, the judge observes at most five sample result rows and systematically rejects empty results and duplicate results. These rules may improve filtering efficiency, but they cannot replace semantic correctness validation and may restrict dataset coverage. A paper whose main contribution includes data quality needs a direct audit of the released corpus, with separate error rates for original and rewritten questions.

  1. The current experiments do not sufficiently attribute improvements to the emphasized dataset properties.

The paper primarily attributes differences among the three trained models to the training data. However, SQaLe uses four length buckets and a curriculum emphasizing the tail, SynSQL uses a different three-bucket curriculum, and BIRD uses uniform sampling. The comparison therefore changes both the data and the training sampling strategy. Equal update counts also do not imply equal actual computation, particularly with multi-turn exploration and dynamic resampling.

Furthermore, changing the visible database size during evaluation tests model robustness to distractor tables, but does not independently demonstrate that training on larger schemas causes the advantage. Cross-corpus differences in value lookup may similarly reflect question style, curriculum, or domain differences. More specific attribution requires controlled ablations within SQaLe that vary schema size, table contents, and domain coverage while holding other factors constant.

Finally, the main external evaluations each use 300 questions, while the computational efficiency comparison uses only 100 moderate-difficulty questions from the SQaLe test set. Without confidence intervals, repeated training runs, or broader external efficiency evaluations, claims about generalization and the compute–accuracy frontier should be more cautious.

Questions for authors

The following questions are intended to clarify the current conclusions, and the answers may change my assessment.

  1. Corpus quality: Could you conduct an independent, stratified manual audit of the final released SQaLe data, covering original questions, all seven rewriting styles, and different difficulty levels? For ambiguous questions that omit conditions, how do you determine whether the inherited SQL remains an appropriate answer? Table 2 reports that 95.1% of foreign-key cells are valid after repair. How are the remaining cases handled, and do they affect otherwise valid queries?
  2. Training comparisons and attribution: Do the main results hold when all three corpora use a shared curriculum or uniform sampling? How many distinct questions, SQL queries, and schemas are included in each training pool, and how many tokens and how much computation are actually consumed during training? Could ablations within SQaLe isolate the contributions of large schemas and richer table contents?
  3. Realism and data splits: What proportions of the final databases consist of original tables and LLM-added tables? Have you checked the domain consistency of added tables and whether they are actually used in queries? Does the train–test split account for source repositories, schema versions, and near-duplicate schemas, rather than only different schema identifiers?
  4. Generalization and efficiency: How were the 300 evaluation questions and 100 moderate-difficulty questions sampled, and how much uncertainty is associated with the results? Could you verify the findings on the full BIRD dev set or a larger fixed subset and provide compute–accuracy comparisons on external datasets? How do the FLOPs estimates in Appendix J account for multi-turn prefix caching and the actual number of activated parameters in MoE models?
2 comments
AuthorsOpenReview ↗

Response Part 1/2

Thank you for highlighting how we can enrich our validation and what we should clarify. We address the remarks and questions below.

W1 & Q1 — Corpus quality

While we agree that the LLM judge is aligned with the BIRD Dev set (94.5% alignment) and not the introduced SQaLe dataset, we consider it a strong proxy for assessing the reliability of the LLM judge. To further verify this, we have now also run an independent LLM judge from another model family (Claude Sonnet 5.5) which marks 1 of 75 SQaLe question–SQL pairs as incorrect (none of the 25 hard cases), against 8% of human-written SQL in the BIRD dev gold set, further demonstrating the quality in SQaLe. We also manually reviewed 83 cases where the judge and BIRD-Dev execution accuracy disagreed (Sec. 2.3), and found that the judge correctly judged 52/83 cases.

Although we envision the training of small text-to-SQL models as the main use-case of the SQaLe dataset and not benchmarking them, these assessments confirm that SQaLe provides a fruitful basis for a new benchmark. Importantly, to validate the quality of SQaLe for its intended use of model training, we experimentally demonstrate that a model trained on SQaLe generalizes well to the key benchmarks in the field.

For short-and-ambiguous rewrites in SQaLe, the independent LLM judge judges the reference SQL most plausible in 24 out of 25 cases. We add these iterations to reflect natural underspecification to make models more robust intrinsically or develop clarification pipelines. We also show that this is a valuable enhancement in SQaLe on an external benchmark built for ambiguity in text-to-SQL, AMBROSIA [1]: on 300 ambiguous questions the SQaLe-trained model returns a valid interpretation for 48.3%, against 33.7% for the BIRD-trained and 27.0% for the SynSQL-trained model.

We will incorporate the validation and ambiguity findings in the revision of the paper.

[1] Saparina, Irina, and Mirella Lapata. "Ambrosia: A benchmark for parsing ambiguous questions into database queries." Advances in Neural Information Processing Systems 37 (2024): 90600-90628.

W2 & Q2 — Training comparisons and attribution

Part of the performance differences may indeed come from the curricula, which differ slightly between the corpora. However, enabling such curricula is part of the point of SQaLe. Its wide range of schema and query sizes enables these curricula, which in turn improve the model’s performance as we show in the experiments in Section 4.3.

We agree that controlled ablations on the SQaLe-trained model would surface the value of the different dataset properties and training curricula, but this would be compute intensive. Instead, we inform the design desiderata by prior research and instill and analyze these properties in the dataset. While we are reserved in specific attributions to dataset properties beyond schema sizes, table content and coverage, we believe that the performance and generalization performance of the SQaLe-trained model versus SynSQL and BIRD-train, demonstrates the value of the informed design of SQaLe well.

All three models were trained for 1,800 GRPO steps of 18 questions × 8 rollouts each, on 3 × H100. Training compute was ≈ 160 GPU-hours per model and differed by less than 1.5% between the corpora. We will report the training budget for our experiments in the paper, using the extra page.

Q3 — Realism and data splits

We check for any influence of schema-level splitting rather than repository-level:

  • Our dataset split is at schema level, so none of the test schemas is in the train split. A check of the overlap in source repositories showed that 197 schemas in the test-set share a source repository with a train-split schema. Excluding these from the test set does not reduce the SQaLe-trained model's performance on the test split, so there is no repository-derived domain leakage.
  • Moreover, we validated that no test schema has a near-duplicate in the train split (table-name Jaccard ≥ 0.8: 0 of 423; identical DDL: 0). Test schemas are also not closer to train than train schemas are to each other, and even test schemas from a shared repository are structurally different from their train neighbours (median Jaccard 0.02).

Regarding realism of the augmented database schemas:

  • SQaLe is semi-synthetic by design to establish a dataset at scale for training specialized text-to-SQL models. To achieve this, each SchemaPile schema is extended with LLM-generated tables besides the original tables.
  • It is important to note that the questions are actually anchored in the original real tables in SchemaPile, not the synthetic ones. 59% of the queries in SQaLe also target at least one original table. Moreover, we ground the schema extension in the original schema, illustrated in Figure 2. We also manually reviewed 20 schemas and found that this results in consistency regarding the schema naming style, language etc. of the original tables. We will report this in the expanded paper.
AuthorsOpenReview ↗

Response Part 2/2

Q4 — Generalization and efficiency

Q4.1 Generalization

We have now tested the SQaLe-trained model on the full BIRD dev set and the BIRD-trained model leads by 3.8 pp (53.9% vs 50.1%), which is mainly driven by answer formatting rather than (in)correct answering. That is, on 37 of the 199 BIRD dev questions that only the BIRD-trained model gets right, the SQaLe-trained model returns exactly the gold rows but adds extra columns (for example the count it ranked by), which BIRD's metric scores as wrong. This happens 79 times for the SQaLe-trained model against 24 times for the BIRD-trained one.

If we would score the models accordingly, the two are on-par (55.4% vs 55.5%). This is notable as SQaLe is not trained on the BIRD-Train corpus and is therefore out-of-domain, which demonstrates the generalization power of SQaLe. We keep BIRD's official metric but will add this reflection. Moreover, SQaLe's generalization performance shows across evaluation datasets: on unseen SQaLe schemas (+13.8 pp), Spider 1.0 dev (+13.8 pp), WikiSQL (+9.7 pp), EHRSQL (+11.4 pp).

4.2 Efficiency

FLOPs and prefix caching, and Mixture-of-Experts models:

  • Appendix J counts every call's full prompt, i.e. it gives no prefix-cache credit. That is an upper bound that weighs most on multi-turn agents. We re-rendered every call of the evaluation episodes and applied vLLM's block-level prefix caching: caching removes 58–66% of each 2B student's compute but leaves the Figure 6 conclusion intact.
  • Figure 6 places Qwen3-30B-A3B and gpt-oss-20b at their total parameter counts. We do account for the parameter count by active parameters (3.3B and 3.6B), as their compute falls about 9× and 6× compared to dense models.

The compute–accuracy on external data:

  • You also asked for compute–accuracy on external data: we now report it on six external sets for the four 2B students.
  • With caching, M_SQaLe spends 1.5–2.3× the BIRD-trained model's inference compute per question, and it lies on the accuracy–compute Pareto front of the four on every set except BIRD dev.
  • The advantage is not due to that extra compute though. With its tool budget cut from six turns to three, the SQaLe-trained model uses the same compute as the BIRD-trained one (64 vs 65 TFLOPs per question) and still leads by 10.7 pp on unseen SQaLe schemas.
  • At an equal three-turn budget for both models the lead is 18.6 pp.

We hope that we have addressed your remarks and questions, and that our insights result in a positive assessment-- we are very excited about the SQaLe dataset and the vision it enables. We also welcome any remaining questions you may have.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.