- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a confidence estimation framework for Text-to-SQL that separates generation from validation. A generator LLM writes and executes SQL queries, while a different critic LLM judges the queries and their execution results. A Random Forest fuses multiple confidence signals, and conformal calibration provides finite-sample guarantees on selective risk and false abstention. Experiments on Spider demonstrate improvements over existing baselines, with additional zero-shot transfer evaluations on BIRD and NL2SQL-BUGs.
Critical strengths
- The paper addresses an important problem in Text-to-SQL: detecting incorrect SQL queries and determining when the system should abstain, which is crucial for reliable real-world applications.
- The paper provides finite-sample guarantees on selective risk and false abstention using conformal prediction and Learn-then-Test calibration under appropriate assumptions.
- The proposed method outperforms several existing baselines on Spider. The paper also includes ablation studies, multiple model comparisons, and zero-shot transfer experiments to evaluate its effectiveness.
Critical weaknesses
Lack of Controlled Comparisons Across Models. The authors vary the model family, model size, and training configurations when comparing different generators and critics, making it difficult to identify the factors responsible for performance differences. For example, it is unclear why DeepSeek-V4.1-Flash performs worse than Qwen3.8-27B. More controlled experiments comparing models of different sizes within the same family, as well as similarly sized models across different families, would help clarify these results. 2. Limited Representativeness of the Generator Used for Baseline Comparisons. The authors primarily use Olmo-3-32B-Think for baseline comparisons and ablation studies, which may not adequately represent the capabilities of more recent and stronger Text-to-SQL models. It would be beneficial to repeat the full baseline comparisons using other generators, such as Qwen3.6-35B-A3B, to demonstrate whether the proposed method consistently outperforms existing approaches across different model backbones. 3. Limited Dataset Diversity and Task Complexity. The main experiments are largely conducted on Spider, which is relatively simple compared to more realistic Text-to-SQL benchmarks. Although the authors report zero-shot transfer results on BIRD, they do not provide comprehensive baseline comparisons, and the risk-control guarantees fail to transfer under distribution shift. More extensive evaluations on challenging benchmarks such as BIRD and Spider 2.0 are needed to better assess the method's robustness and practical applicability.
Questions for authors
- Could the authors explain why DeepSeek-V4.1-Flash underperforms Qwen3.8-27B despite their different model capabilities and configurations? Could the authors provide more controlled comparisons, such as models of different sizes within the same family or similarly sized models across different families, to better understand the impact of model selection?
- Would the proposed method maintain its performance advantages when evaluated against the same baselines using stronger generators?
- Could the authors provide more comprehensive baseline comparisons on BIRD and evaluate the proposed method on more challenging benchmarks, such as Spider 2.0?
4 comments
Response (Part 2/2): BIRD baselines, Spider 2.0-lite, Beaver and risk control on new benchmarks (Q3)
Q3. Could the authors provide more comprehensive baseline comparisons on BIRD and evaluate the proposed method on more challenging benchmarks, such as Spider 2.0?
We centred on Spider because most prior work uses it and it gives enough calibration queries (636) for the risk bound. To answer your question, we evaluated on BIRD and Spider 2.0-lite. Ours is the full 41 feature forest, and logit-free keeps only its 5 LLM reported scores (paper's Table 5). Zero shot forests are trained on Spider, in-domain ones on the target. Cells give zero shot AUARC, in-domain in brackets. Random and perfect ranking give and for accuracy .
(a) BIRD, same generator and dialect as Spider. 1,534 dev queries (GPT-OSS-120B, Gemma critic), 34.2% correct, in-domain on 60/20/20 splits over 10 seeds. We add SIMBA UQ (Bhattacharjya et al., 2025), reported on Spider and BIRD.
| Method | Calls | BIRD |
|---|---|---|
| Random / perfect | 34.2 / 70.9 | |
| SAC-Product / Pooled | 1 | 36.0 / 37.7 |
| Single CoT Verify | 2 | 46.5 |
| Self-consistency | 11 | 44.7 |
| SIMBA UQ | 11 | 46.0 (47.1) |
| SAC + Pooled, trained | 1 | 44.1 (44.7) |
| Ours (all 41 features) | 2 | 51.5 (54.6) |
| Ours, logit-free (5 of 41) | 2 | 51.0 (53.7) |
Ours is best in both settings. In-domain it leads SIMBA by 7.5 AUARC and 9.5 AUROC (78.5 against 69.0) and closes , or 56%, of the random to perfect range, against 35% for SIMBA and 64% for ours on Spider (Q2). It needs only 2 LLM calls against 11.
(b) Spider 2.0-lite, a new dialect. On Spider 2.0-lite (Lei et al., 2025) we ran 339 of 547 tasks (135 SQLite, 204 BigQuery), 29.5% correct. Snowflake (207 tasks) needs Spider 2.0 team approval, difficult to obtain during the rebuttal period, and one BigQuery dataset is no longer public. In-domain uses 5 fold cross validation, and self consistency 3 samples (free BigQuery quota).
| Method | Calls | Spider 2.0-lite |
|---|---|---|
| Random / perfect | 29.5 / 65.5 | |
| SAC-Product / Pooled | 1 | 32.5 / 33.6 |
| Single CoT Verify | 2 | 45.2 |
| Self-consistency | 4 | 38.7 |
| SIMBA UQ | 11 | 39.6 (39.0) |
| SAC + Pooled, trained | 1 | 30.8 (33.7) |
| Ours (all 41 features) | 2 | 40.4 (43.0) |
| Ours, logit-free (5 of 41) | 2 | 47.9 (47.9) |
The logit-free variant is best in both settings, significantly above SIMBA (+8.2) and self consistency (+9.0) in a paired bootstrap, and 2.6 above Single CoT Verify, not significant. On BEAVER (Chen et al., 2024), where the best of four generators answers only 10.8% of 595 questions, it reaches AUROC 73.9 zero shot, but with 64 correct queries the 95% intervals (about ±5 AUARC) cannot separate it from 11 call self consistency (18.8 against 19.5). We recommend the full model when the generator and SQL dialect match training (BIRD), and the logit-free variant when either changes or labels are few (Spider 2.0-lite, BEAVER), since token probabilities learned on Spider only partly carry over.
(c) Risk control on a new benchmark. A Spider certified threshold does not carry over to BIRD (Section 5.6). No estimator can do this without target labels (Tibshirani et al., 2019, Barber et al., 2023). A few hundred BIRD labels restore it, and the binomial test sets how many. At (66% of BIRD queries are wrong):
| BIRD calibration | Certified | Coverage | Test risk |
|---|---|---|---|
| 614 queries (40/40/20) | 10/10 | 27.9% | 37.4% |
| Spider forest, threshold refit on 600 | 82/100 | 19.2% | 38.7% |
Every certified run kept its test risk below .
Where this sits in the literature. Prior Text-to-SQL estimators use one signal family: token probabilities (Ramachandran & Sarawagi, 2024, Entezari Maleki et al., 2026), sub-clause frequencies (Liu et al., 2025) or sample agreement (Wang et al., 2023, Bhattacharjya et al., 2025). SQLens and SQLFixAgent detect or repair errors without a calibrated score, and Chen et al. (2025) certify abstention only in schema linking. None combines a second model validating the executed result, fusion with the generator's signals, and a finite sample bound on accepted-query error.
What we will add. These tables go into Section 5.6 and Limitation 4, and we add the label requirement to Proposition 2. We hope this addresses your concerns.
References (others are in the paper)
- Tibshirani et al. (2019). Conformal Prediction Under Covariate Shift. NeurIPS.
- Barber et al. (2023). Conformal Prediction Beyond Exchangeability. Annals of Statistics.
- Bhattacharjya et al. (2025). SIMBA UQ: Similarity-Based Aggregation for Uncertainty Quantification in Large Language Models. Findings of EMNLP.
Thanks for your feedback. My questions are partially addressed. It is better to evaluated the complete Spider 2.0 through major revision since it is more challenge and practical text2SQL task.
Spider 2.0 evaluation
Thank you for reading our response and for the follow up. We agree that Spider 2.0 is the more realistic test. Its current release has two settings, and our method fits one of them directly.
Spider 2.0-Snow (547 tasks). These are the same 547 questions as Spider 2.0-lite, with every database hosted on Snowflake. Each task asks for one SQL query whose result is compared with the gold result, which is exactly the setting of our method: the generator writes and executes the query, the critic reviews the query and its result, and the forest scores it. We have already run 339 of these questions in their Spider 2.0-lite form (135 SQLite and 204 BigQuery tasks). The remaining 208 need Snowflake. Our pipeline already supports the Snowflake dialect, so completing the set needs no change to the method, only Snowflake access from the Spider 2.0 team. We will add the full 547-task results to the final version, and if access comes through during the discussion period, we will post them here.
Spider 2.0-DBT (68 tasks). This setting replaced the original agentic Spider 2.0 tasks. Each task is a dbt project in which an agent writes several SQL models over many steps, and it is scored by the final state of the database. There is no single query and result for a critic to review, or for the system to accept or abstain on, so per-query confidence and our risk bound do not apply as they stand. Confidence for multi-step agent workflows is a different and open problem, and 68 tasks are also too few to calibrate a risk threshold, which needs a few hundred labelled queries. We will state this scope in the limitations and name it as future work.
On the 339 questions we could run, our logit-free variant is the best estimator, significantly above SIMBA and self-consistency. Adding the remaining Snowflake questions extends the same table with the same protocol rather than changing the method. We hope this clarifies the scope, and we would be glad to provide any further results.
Response (Part 1/2): Controlled critic comparison (Q1) and stronger generators (Q2)
Thank you for the careful review and concrete suggestions. We are glad you found the problem important and the guarantees and ablations useful. Due to the 5,000-character limit, we answer in two parts: Q1 and Q2 here, and Q3 and prior work in Part 2.
The paper's Table 2 tests whether the estimator depends on one critic (AUARC 89.9 to 92.2 across critics), but family, size and thinking change together there. To answer your question, we ran the controlled comparison you suggested.
Controlled comparison. Only the critic changes, with generator (GPT-OSS-120B), Spider queries (n=3,179), prompt, features and forest fixed. We added Nemotron-3-Ultra (same family as Nemotron-3-Super, 4.6× larger), GPT-OSS-20B, and Gemma-4-31B with thinking off. On the same 10 resampled splits, splits won counts the splits where the first critic has higher AUARC, and p is a paired t-test on the per-split differences.
Within a family, 4.6× more parameters gives no significant gain, while at similar size the family makes a consistent difference. Thinking adds only 0.1.
Why DeepSeek-V4.1-Flash is behind Qwen3.8-27B. It is not the weaker judge. It rejects 20.4% of wrong queries, twice as many as Qwen3.8, but its confidence is less graded. Let s+ and s− be the critic scores of a random correct and a random wrong query. AUROC counts ties as one half, so AUROC=P(s+>s−)+21P(s+=s−)≤1−21P(s+=s−). Verbalized confidence sits on a few high values (Xiong et al., 2024), and ties cannot be ranked. On all 3,179 queries:
So more of DeepSeek's pairs tie, and it ranks worse alone (72.4 against 75.7). Inside our estimator the gap is only 0.7 AUARC, and the six critics span 1.7 AUARC (10-split means 89.7 to 91.4) against 18 AUROC points alone (57.5 to 75.7), as the generator's signals separate queries a critic ties. Recalibrating one score cannot do this, since a monotone map keeps ties tied. A critic should therefore be chosen for graded confidence, not size.
While the paper's Table 4 compares all methods on Olmo-3-32B so that differences come from the estimator, its Appendix Table 11 varies the backbone for our method only. To answer your question, we tested two stronger generators.
Experiment. We reran the paper's Table 4 on GPT-OSS-120B (77.3% correct) and Qwen3.8-27B (83.4%), next to Olmo-3-32B (73.9%). Qwen3.6-35B-A3B is not on our gateway, so we used Qwen3.8-27B from the same family. All methods score the same queries on the same 10 splits, self-consistency uses K=10, and the logit baselines also get a version trained with our forest. Values are 10-split AUARC, so Olmo values differ slightly from the paper.
Yes, the advantage holds. Ours is best on all three generators, 5.9, 3.5 and 3.8 AUARC above the best baseline. Normalized by the room above random, (AUARC−ρ)/(ρln(1/ρ)) for base accuracy ρ, ours keeps 0.65, 0.64 and 0.73 against 0.39, 0.46 and 0.48, so it does not shrink on stronger generators. The reason is fusion: each logit baseline relies on one internal signal that swings with the backbone (AUROC 48 to 53 on GPT-OSS-120B, 65 to 66 on Qwen3.8), while our forest combines several. Without the critic it already reaches 87.1, 89.2 and 93.9. The critic adds 0.5 to 1.4 and lifts certified coverage on GPT-OSS-120B at ε=0.15 from 42.6% to 52.4%.
What we will add. This table goes into the paper's Table 4, and the Q1 tables and tie analysis go in the appendix, replacing the Section 5.2 sentence on critic differences.
Please see Part 2 for Q3, prior work and references.