- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduced SQALE, a large-scale semi-synthetic text-to-SQL dataset (1.4M questions + 176k queries) built on 9k real-world schemas extracted from GitHub. Defined the critieria used to design the generation pipline, including schema expansion, value synthesis, question generation, etc. Followed by some analysis around the dataset, e.g., testing results with model trained on different datasets. The results shows SLM trained on SQaLe achieves good performance.
Critical strengths
Strengths
S1. The paper introduces a large-scale text-to-SQL dataset, including 1.4M questions and 176k SQL queries, based on seed schemas obtained from real-world problems.
S2. Well-defined design criteria and structured generation pipeline. The paper listed 12 dataset design criteria covering schema-level, data-level, and query-level properties. This definition improves the clarity.
Critical weaknesses
W1. Limited quality guaruntee. Only one human validation was done in the paper, judge verdicts against human labels, on 225 judge calls by one author. Moreover, it's done on BIRD-dev not SQaLe, leaving a potential distribution gap. The reliability of the dataset is then questioned, especially for complex queries, and underspecified paraphrases (ambiguous).
W2. The performance improvements might be biased and not significant on BIRD. I'm not convinced whether the improvement can be attributed specifically to the dataset improvement, rather than the test distribution simlarity, training curriculum, etc. Other tests suggests that the gains on SQaLe may not translate uniformly to external benchmarks. and achieve relative similar performance on both BIRD and EHRSQL, considering SQaLe has way more training samples and BIRD-test is relatively simple.
Questions for authors
Q1. How does the training curriculum affect the model training with SQaLe, does it require careful design and a bunch of testing? Is this GRPO training stable?
Q2. How's the model performance on complex queries with human annotated dataset, like Spider 2.0
Q3. Has the author trained models with data from different difficulty level, and how does the complexity affects the performance across levels.
Q4. What's the cost of data creation, and model training cost?
Q5. Can the authors share some examples of the ambiguous queries, and what is expected to handle these queries.
4 comments
Response Part 2/2
Q3 Influence difficulty levels We trained on all difficulty levels jointly, without per-level ablations; the length curriculum (Q1) shifts training towards hard questions over time, as the two long buckets are mostly labelled hard (59% and 76%). On 420 unseen SQaLe test questions (140 per level), the SQaLe-trained model scores 69.3 / 48.6 / 30.7% on simple / moderate / hard (base model: 40.0 / 20.0 / 12.1%). Its advantage over the other trained models is largest at moderate (+23.6 pp) and still +11.8 pp at hard. We will add this breakdown to the paper.
Q4 Cost of data generation and training In H100 GPU-hours (GPUs allocated × wall-clock time):
- Data creation: all LLM calls ran on self-hosted open-weight Qwen models (Appendix M), so there were no API costs. The 241 generation shards (2 GPUs each) took 1,432 GPU-hours, including ~10% on tasks that timed out or crashed.
- Training: one GRPO run of the 2B model (3 × H100 for 50 h) costs 150 GPU-hours, identical for the SQaLe-, BIRD- and SynSQL-trained models.
We will report these budgets in the paper in the appendix.
Q5 Ambiguous query style examples
- Original: "Find the queue names and their maximum job limits for all queues associated with compute resources that have a maximum memory per node greater than 256 GB."
- Short and ambiguous: "Need queue names and job limits for the heavy-memory compute setups."
- Reference SQL:
SELECT bq.QUEUE_NAME, bq.MAX_JOB_IN_QUEUE FROM BATCH_QUEUE bq JOIN COMPUTE_RESOURCE cr ON bq.COMPUTE_RESOURCE_ID = cr.RESOURCE_ID WHERE cr.MAX_MEMORY_NODE > 256;
Here, "heavy-memory" leaves the 256 GB threshold implicit.
With the schema withheld, the agent is expected to resolve such underspecification by exploring tables, values and join paths, and to answer on a best-effort basis. Keeping a single reference SQL is a deliberate scoping decision, as our setup has no clarification channel.
Beyond best-effort answering, we expect these questions to be used for building disambiguation pipelines. Every ambiguous rewrite is labelled with its style and paired with its fully specified original and the shared reference SQL, so the difference between the two questions is exactly the information a clarification step has to recover (here, the 256 GB threshold). This provides supervision for detecting underspecified requests, asking the right clarification question, and mapping the clarified request to SQL; users can also select or exclude the style. The style is already useful in its current form: on the ambiguity benchmark AMBROSIA [1], the SQaLe-trained model returns a valid interpretation for 48.3% of 300 ambiguous questions, against 33.7% for the BIRD-trained and 27.0% for the SynSQL-trained model. We will add examples of all seven styles to the appendix.
[1] Saparina, Irina, and Mirella Lapata. "Ambrosia: A benchmark for parsing ambiguous questions into database queries." Advances in Neural Information Processing Systems 37 (2024): 90600-90628.
We hope that we have addressed your remarks and questions and that our response positively contributes to your assessment: we are very excited about the SQaLe dataset and how it materializes the potential for training small specialized text-to-SQL models. We also welcome any remaining questions you may have.
Thank you for your response. The testing or human evaluation of LLM judges can not serve as a definitive ground truth validation of the dataset, and sample size are not enough to support the claim. I'll maintain my score.
Thank you for the follow-up. We agree that validating the judge is not a definitive check for the ground-truth accuracy of the dataset and we do not claim that SQaLe is error-free. What we show in the paper is that SQaLe is a reliable training corpus and a better basis for training small text-to-SQL models than prior datasets. Like other training corpora such as FineWeb [1], we establish this through controlled training comparisons. All models are trained from the same base model and use the same number of training steps, batch size and hyperparameters, apart from the curriculum schedule. To that end, the full evaluation results on external benchmarks show that performs best in 9 of 10 evaluations. The only exception to this is the BIRD dev dataset, where has an in-distribution advantage from training on BIRD’s own training split.
Full results:
| Benchmark | Questions | Base (Qwen3.5-2B) | |||
|---|---|---|---|---|---|
| Unseen SQaLe test schemas | 420 | 49.5 | 35.7 | 29.5 | 24.0 |
| BIRD dev (full) | 1,532 | 50.1 | 53.9 | 41.4 | 28.3 |
| Spider 1.0 dev | 298 | 76.8 | 63.1 | 64.4 | 40.9 |
| Spider-Syn | 298 | 69.1 | 59.4 | 60.4 | 38.3 |
| Spider-Realistic | 298 | 71.8 | 62.1 | 61.1 | 43.3 |
| WikiSQL | 300 | 66.7 | 57.0 | 37.0 | 26.7 |
| EHRSQL (all) | 300 | 25.3 | 21.0 | 15.0 | 7.0 |
| EHRSQL (gold not a single NULL/0 row) | 210 | 30.5 | 19.0 | 12.9 | 5.7 |
| AMBROSIA, ambiguous questions (any valid reading) | 300 | 48.3 | 33.7 | 27.0 | 13.0 |
| Spider 2.0-lite, local SQLite subset (mean of 3 seeds) | 75 | 5.8 | 4.9 | 2.2 | 1.3 |
As described above, we also
- build quality control into the dataset generation as every pair must execute against the database and be accepted by an LLM judge that we validated against human labels.
- perform independent post-hoc checks of SQaLe with a judge from a different model family that rates the instances in SQaLe as at least as accurate as human-annotated instances in the BIRD dataset. We consider this a more informative analysis than comparing against human labels because human-written gold SQL is not a clean reference in complex text-to-SQL settings. [2] find substantial annotation errors of 52.8% in BIRD dev and 66.1% in Spider 2.0-Snow, a result that is in line with per-domain findings by [3]. These error analyses consistently show that a large share of supposed model failures turn out to be correct SQL scored against wrong or overly strict gold labels [4, 2], which is in line with our manual evaluation of our judge that correctly disagreed with BIRD’s gold SQL in 52/83 cases (Sec 2.3). LLM-based reviewers have also been shown to surface annotation errors that human annotation misses and proposed as effective tools in creating robust datasets [2].
[1] Penedo et al. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. NeurIPS 2024 Datasets and Benchmarks Track.
[2] T. Jin, Y. Choi, Y. Zhu, D. Kang. Text-to-SQL Benchmarks are Broken: An In-Depth Analysis of Annotation Errors. CIDR 2026. (Extended version: Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards, arXiv:2601.08778)
[3] N. Wretblad, F. Riseby, R. Biswas, A. Ahmadi, O. Holmström. Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark. ACL 2024.
[4] M. Pourreza, D. Rafiei. Evaluating Cross-Domain Text-to-SQL Models and Benchmarks. EMNLP 2023.
Response Part 1/2
Thank you for the constructive feedback and questions, which we gladly address below.
W1 – Quality validation
We consider BIRD-Dev a strong proxy for assessing the reliability of the LLM judge as it provides us with a relatively large set of human-generated SQL queries. We further verify verify the quality of the dataset with a strong LLM judge from another model family (Claude Sonnet 5.5) which marks 1 of 75 SQaLe question–SQL pairs as incorrect (none of the 25 hard cases), against 8% of human-written SQL in the BIRD dev gold set.
Although we envision the training of small text-to-SQL models as the main use-case of the SQaLe dataset and not benchmarking them, these assessments confirm that SQaLe provides a fruitful basis for a new benchmark. Importantly, to validate the quality of SQaLe for its intended use of model training, we experimentally demonstrate that a model trained on SQaLe generalizes well to the key benchmarks in the field.
For short-and-ambiguous rewrites in SQaLe, the independent LLM judge judges the reference SQL most plausible in 24 out of 25 cases. We further assess the quality utility by downstream evaluation on an ambiguity benchmark, AMBROSIA [1]. The SQaLe-trained model returns a valid interpretation of an ambiguous question for 48.3% of 300 questions, against 33.7% for the BIRD-trained model (+14.7 pp), without losing accuracy relative to the unambiguous rewrites.
W2 – Model performance evaluation
The SQaLe-trained model does indeed not outperform the BIRD-trained model on BIRD dev (52.3 vs 54.7). While the BIRD-trained model is of course in-domain and the SQaLe-model generalizes relatively well out-domain, our manual analysis shows that the gap is mainly driven by answer formatting rather than incorrect answering. That is, on 37 of the 199 BIRD dev questions that only the BIRD-trained model gets right, the SQaLe-trained model returns exactly the gold rows but adds extra columns (for example the count it ranked by), which BIRD's metric scores as incorrect. This happens 79 times for the SQaLe-trained model against 24 times for the BIRD-trained one.
If we would score the models accordingly, the two are on-par (55.4% vs 55.5%). This is a notable result, as SQaLe is not trained on the BIRD-Train corpus and is therefore out-of-domain, which demonstrates the generalization power of SQaLe. We keep BIRD's official metric but will add this reflection. Moreover, SQaLe's generalization performance shows across evaluation datasets: on unseen SQaLe schemas (+13.8 pp), Spider 1.0 dev (+13.8 pp), WikiSQL (+9.7 pp), EHRSQL (+11.4 pp).
Questions
Q1 Influence training curriculum The curriculum is a fixed schedule, not tuned per run (Appendix H): questions are bucketed by gold-SQL length (50th/80th/95th percentiles), the first 400 steps draw only short questions, and the mixture then anneals linearly towards the long tail over 1,000 steps, with a floor of 0.15 per bucket so short queries keep contributing gradient. In earlier runs, this notably improved performance on hard and long queries, whose signal is otherwise averaged away by the many short ones (half of SQaLe's gold queries are under 150 characters). Starting close to the natural mixture matters for GRPO, since groups in which all candidates fail carry no advantage signal.
After initial instabilities, training was stabilized by (Appendix I): token-level PPO clipping with DAPO clip-higher (ε = 0.2 / 0.28) and an importance-ratio cap of 2.0; a k3 KL penalty adapted to a target of 0.005, with the reference re-anchored every 111 updates; Dr. GRPO-style advantages clipped to ±2 with negatives weighted 0.7; DAPO resampling of zero-variance groups; entropy-triggered guards on the KL weight and learning rate; and on-policy updates at temperature 1.0 with gradient clipping, bounded rewards and episode caps. All three models use these identical hyperparameters, apart from the curriculum strength.
Q2 Complex queries as in Spider2
We evaluated all models on the part of Spider 2.0-lite that runs on local SQLite files (75 of 547 instances; the rest need BigQuery, Snowflake or DuckDB), with three decoding seeds and Spider 2.0's own scorer. All 2B models are at the floor: SQaLe-trained 5.8%, BIRD-trained 4.9%, SynSQL-trained 2.2%, base 1.3%. These enterprise tasks are beyond a 2B agent in this protocol for every training corpus, which we will discuss as future work.
On the human-annotated Spider 1.0 dev set, the SQaLe-trained model is clearly the most accurate (76.8% vs 63.1% for the BIRD-trained model, +13.8 pp, 298 questions), also under perturbation: Spider-Syn 69.1% vs 59.4%, Spider-Realistic 71.8% vs 62.1%.