acceptodds
Under review as a conference paper at ICLR 2027

Know When to Abstain: Conformal Confidence Estimation for Text-to-SQL

Abstract

Text-to-SQL systems often return queries that execute successfully but answer the wrong question. Detecting these errors at deployment is hard because execution correctness is defined against a gold query, which exists only in labeled data. We formulate confidence estimation for Text-to-SQL as probabilistic query validation under a latent correctness variable that is observable at calibration time through gold-SQL execution and latent at deployment. Our estimator separates generation from validation. A generator LLM writes and executes the query, and a different critic LLM judges the query and its execution result. The critic's validation and execution-grounded alignment scores are fused with the generator's verbalized confidence, token-level uncertainty, and SQL-structural features in a Random Forest, using two sequential LLM calls. Two calibration steps give finite-sample guarantees. A Learn-then-Test threshold ensures, with probability 1 − δ, that the error rate among accepted queries stays below a user-chosen budget. A conformal p-value bounds the false abstention rate. On Spider we use GPT-OSS-120B as generator and Gemma-4-31B as critic. The estimator obtains an AUARC of 0.899. Without abstention, 22.6% of queries are wrong. With a 15% risk budget (δ = 0.1), a threshold is certified on all 10 resampled splits, and the system accepts 52.4% of queries on average with 9.1% observed error. Five different critic models give similar results. In a controlled comparison on Spider with Olmo-3-32B as generator, the estimator obtains an AUARC of 89.5, compared with 81.2 for the strongest logit-based baseline. Applied to BIRD without retraining, the confidence scores remain informative (AUROC 0.72, against 0.79 on Spider), but the risk guarantee does not transfer, as expected without exchangeability.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Text-to-SQL, Uncertainty Quantification, Conformal Prediction, Confidence Estimation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KNh3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a confidence estimation framework for Text-to-SQL that separates generation from validation. A generator LLM writes and executes SQL queries, while a different critic LLM judges the queries and their execution results. A Random Forest fuses multiple confidence signals, and conformal calibration provides finite-sample guarantees on selective risk and false abstention. Experiments on Spider demonstrate improvements over existing baselines, with additional zero-shot transfer evaluations on BIRD and NL2SQL-BUGs.

Critical strengths
  1. The paper addresses an important problem in Text-to-SQL: detecting incorrect SQL queries and determining when the system should abstain, which is crucial for reliable real-world applications.
  2. The paper provides finite-sample guarantees on selective risk and false abstention using conformal prediction and Learn-then-Test calibration under appropriate assumptions.
  3. The proposed method outperforms several existing baselines on Spider. The paper also includes ablation studies, multiple model comparisons, and zero-shot transfer experiments to evaluate its effectiveness.
Critical weaknesses

Lack of Controlled Comparisons Across Models. The authors vary the model family, model size, and training configurations when comparing different generators and critics, making it difficult to identify the factors responsible for performance differences. For example, it is unclear why DeepSeek-V4.1-Flash performs worse than Qwen3.8-27B. More controlled experiments comparing models of different sizes within the same family, as well as similarly sized models across different families, would help clarify these results. 2. Limited Representativeness of the Generator Used for Baseline Comparisons. The authors primarily use Olmo-3-32B-Think for baseline comparisons and ablation studies, which may not adequately represent the capabilities of more recent and stronger Text-to-SQL models. It would be beneficial to repeat the full baseline comparisons using other generators, such as Qwen3.6-35B-A3B, to demonstrate whether the proposed method consistently outperforms existing approaches across different model backbones. 3. Limited Dataset Diversity and Task Complexity. The main experiments are largely conducted on Spider, which is relatively simple compared to more realistic Text-to-SQL benchmarks. Although the authors report zero-shot transfer results on BIRD, they do not provide comprehensive baseline comparisons, and the risk-control guarantees fail to transfer under distribution shift. More extensive evaluations on challenging benchmarks such as BIRD and Spider 2.0 are needed to better assess the method's robustness and practical applicability.

Questions for authors
  1. Could the authors explain why DeepSeek-V4.1-Flash underperforms Qwen3.8-27B despite their different model capabilities and configurations? Could the authors provide more controlled comparisons, such as models of different sizes within the same family or similarly sized models across different families, to better understand the impact of model selection?
  2. Would the proposed method maintain its performance advantages when evaluated against the same baselines using stronger generators?
  3. Could the authors provide more comprehensive baseline comparisons on BIRD and evaluate the proposed method on more challenging benchmarks, such as Spider 2.0?
4 comments
AuthorsOpenReview ↗

Response (Part 1/2): Controlled critic comparison (Q1) and stronger generators (Q2)

Thank you for the careful review and concrete suggestions. We are glad you found the problem important and the guarantees and ablations useful. Due to the 5,000-character limit, we answer in two parts: Q1 and Q2 here, and Q3 and prior work in Part 2.

Q1. Could the authors explain why DeepSeek-V4.1-Flash underperforms Qwen3.8-27B despite their different model capabilities and configurations? Could the authors provide more controlled comparisons, such as models of different sizes within the same family or similarly sized models across different families, to better understand the impact of model selection?

The paper's Table 2 tests whether the estimator depends on one critic (AUARC 89.9 to 92.2 across critics), but family, size and thinking change together there. To answer your question, we ran the controlled comparison you suggested.

Controlled comparison. Only the critic changes, with generator (GPT-OSS-120B), Spider queries (n=3,179n=3{,}179), prompt, features and forest fixed. We added Nemotron-3-Ultra (same family as Nemotron-3-Super, 4.6× larger), GPT-OSS-20B, and Gemma-4-31B with thinking off. On the same 10 resampled splits, splits won counts the splits where the first critic has higher AUARC, and p is a paired t-test on the per-split differences.

FactorComparisonAUARC gainSplits wonp
SizeNemotron Ultra vs Super+0.47/100.066
FamilyQwen3.8-27B vs Gemma-4-31B+1.410/10<0.001
ThinkingGemma on vs off+0.19/100.010
Your pairQwen3.8-27B vs DeepSeek-V4.1+0.79/100.007

Within a family, 4.6× more parameters gives no significant gain, while at similar size the family makes a consistent difference. Thinking adds only 0.1.

Why DeepSeek-V4.1-Flash is behind Qwen3.8-27B. It is not the weaker judge. It rejects 20.4% of wrong queries, twice as many as Qwen3.8, but its confidence is less graded. Let s+s^+ and s−s^- be the critic scores of a random correct and a random wrong query. AUROC counts ties as one half, so AUROC=P(s+>s−)+12P(s+=s−)≤1−12P(s+=s−).\mathrm{AUROC}=P(s^+>s^-)+\frac12P(s^+=s^-)\le 1-\frac12P(s^+=s^-). Verbalized confidence sits on a few high values (Xiong et al., 2024), and ties cannot be ranked. On all 3,179 queries:

CriticDistinct valuesP(s+=s−)P(s^+=s^-)Critic alone, AUROC
Qwen3.8-27B170.1975.7
DeepSeek-V4.1100.2972.4
Gemma-4-31B40.8157.5

So more of DeepSeek's pairs tie, and it ranks worse alone (72.4 against 75.7). Inside our estimator the gap is only 0.7 AUARC, and the six critics span 1.7 AUARC (10-split means 89.7 to 91.4) against 18 AUROC points alone (57.5 to 75.7), as the generator's signals separate queries a critic ties. Recalibrating one score cannot do this, since a monotone map keeps ties tied. A critic should therefore be chosen for graded confidence, not size.

Q2. Would the proposed method maintain its performance advantages when evaluated against the same baselines using stronger generators?

While the paper's Table 4 compares all methods on Olmo-3-32B so that differences come from the estimator, its Appendix Table 11 varies the backbone for our method only. To answer your question, we tested two stronger generators.

Experiment. We reran the paper's Table 4 on GPT-OSS-120B (77.3% correct) and Qwen3.8-27B (83.4%), next to Olmo-3-32B (73.9%). Qwen3.6-35B-A3B is not on our gateway, so we used Qwen3.8-27B from the same family. All methods score the same queries on the same 10 splits, self-consistency uses K=10K=10, and the logit baselines also get a version trained with our forest. Values are 10-split AUARC, so Olmo values differ slightly from the paper.

MethodCallsOlmo-3-32BGPT-OSS-120BQwen3.8-27B
SAC-Product / Pooled179.3 / 80.778.6 / 81.489.1 / 89.3
Self-consistency, K=10K=1011n/a84.089.9
SAC + Pooled in our forest182.686.590.6
Ours288.590.094.4

Yes, the advantage holds. Ours is best on all three generators, 5.9, 3.5 and 3.8 AUARC above the best baseline. Normalized by the room above random, (AUARC−ρ)/(ρln⁡(1/ρ))(\mathrm{AUARC}-\rho)/(\rho\ln(1/\rho)) for base accuracy ρ\rho, ours keeps 0.65, 0.64 and 0.73 against 0.39, 0.46 and 0.48, so it does not shrink on stronger generators. The reason is fusion: each logit baseline relies on one internal signal that swings with the backbone (AUROC 48 to 53 on GPT-OSS-120B, 65 to 66 on Qwen3.8), while our forest combines several. Without the critic it already reaches 87.1, 89.2 and 93.9. The critic adds 0.5 to 1.4 and lifts certified coverage on GPT-OSS-120B at ε=0.15\varepsilon=0.15 from 42.6% to 52.4%.

What we will add. This table goes into the paper's Table 4, and the Q1 tables and tie analysis go in the appendix, replacing the Section 5.2 sentence on critic differences.

Please see Part 2 for Q3, prior work and references.

AuthorsOpenReview ↗

Response (Part 2/2): BIRD baselines, Spider 2.0-lite, Beaver and risk control on new benchmarks (Q3)

Q3. Could the authors provide more comprehensive baseline comparisons on BIRD and evaluate the proposed method on more challenging benchmarks, such as Spider 2.0?

We centred on Spider because most prior work uses it and it gives enough calibration queries (636) for the risk bound. To answer your question, we evaluated on BIRD and Spider 2.0-lite. Ours is the full 41 feature forest, and logit-free keeps only its 5 LLM reported scores (paper's Table 5). Zero shot forests are trained on Spider, in-domain ones on the target. Cells give zero shot AUARC, in-domain in brackets. Random and perfect ranking give ρρ and ρ(1+ln⁡(1/ρ))ρ(1+\ln(1/ρ)) for accuracy ρρ.

(a) BIRD, same generator and dialect as Spider. 1,534 dev queries (GPT-OSS-120B, Gemma critic), 34.2% correct, in-domain on 60/20/20 splits over 10 seeds. We add SIMBA UQ (Bhattacharjya et al., 2025), reported on Spider and BIRD.

MethodCallsBIRD
Random / perfect34.2 / 70.9
SAC-Product / Pooled136.0 / 37.7
Single CoT Verify246.5
Self-consistency1144.7
SIMBA UQ1146.0 (47.1)
SAC + Pooled, trained144.1 (44.7)
Ours (all 41 features)251.5 (54.6)
Ours, logit-free (5 of 41)251.0 (53.7)

Ours is best in both settings. In-domain it leads SIMBA by 7.5 AUARC and 9.5 AUROC (78.5 against 69.0) and closes (54.6−34.2)/(70.9−34.2)(54.6-34.2)/(70.9-34.2), or 56%, of the random to perfect range, against 35% for SIMBA and 64% for ours on Spider (Q2). It needs only 2 LLM calls against 11.

(b) Spider 2.0-lite, a new dialect. On Spider 2.0-lite (Lei et al., 2025) we ran 339 of 547 tasks (135 SQLite, 204 BigQuery), 29.5% correct. Snowflake (207 tasks) needs Spider 2.0 team approval, difficult to obtain during the rebuttal period, and one BigQuery dataset is no longer public. In-domain uses 5 fold cross validation, and self consistency 3 samples (free BigQuery quota).

MethodCallsSpider 2.0-lite
Random / perfect29.5 / 65.5
SAC-Product / Pooled132.5 / 33.6
Single CoT Verify245.2
Self-consistency438.7
SIMBA UQ1139.6 (39.0)
SAC + Pooled, trained130.8 (33.7)
Ours (all 41 features)240.4 (43.0)
Ours, logit-free (5 of 41)247.9 (47.9)

The logit-free variant is best in both settings, significantly above SIMBA (+8.2) and self consistency (+9.0) in a paired bootstrap, and 2.6 above Single CoT Verify, not significant. On BEAVER (Chen et al., 2024), where the best of four generators answers only 10.8% of 595 questions, it reaches AUROC 73.9 zero shot, but with 64 correct queries the 95% intervals (about ±5 AUARC) cannot separate it from 11 call self consistency (18.8 against 19.5). We recommend the full model when the generator and SQL dialect match training (BIRD), and the logit-free variant when either changes or labels are few (Spider 2.0-lite, BEAVER), since token probabilities learned on Spider only partly carry over.

(c) Risk control on a new benchmark. A Spider certified threshold does not carry over to BIRD (Section 5.6). No estimator can do this without target labels (Tibshirani et al., 2019, Barber et al., 2023). A few hundred BIRD labels restore it, and the binomial test sets how many. At ε=0.5\varepsilon=0.5 (66% of BIRD queries are wrong):

BIRD calibrationCertifiedCoverageTest risk
614 queries (40/40/20)10/1027.9%37.4%
Spider forest, threshold refit on 60082/10019.2%38.7%

Every certified run kept its test risk below ε\varepsilon.

Where this sits in the literature. Prior Text-to-SQL estimators use one signal family: token probabilities (Ramachandran & Sarawagi, 2024, Entezari Maleki et al., 2026), sub-clause frequencies (Liu et al., 2025) or sample agreement (Wang et al., 2023, Bhattacharjya et al., 2025). SQLens and SQLFixAgent detect or repair errors without a calibrated score, and Chen et al. (2025) certify abstention only in schema linking. None combines a second model validating the executed result, fusion with the generator's signals, and a finite sample bound on accepted-query error.

What we will add. These tables go into Section 5.6 and Limitation 4, and we add the label requirement to Proposition 2. We hope this addresses your concerns.

References (others are in the paper)

Thanks for your feedback. My questions are partially addressed. It is better to evaluated the complete Spider 2.0 through major revision since it is more challenge and practical text2SQL task.

AuthorsOpenReview ↗

Spider 2.0 evaluation

Thank you for reading our response and for the follow up. We agree that Spider 2.0 is the more realistic test. Its current release has two settings, and our method fits one of them directly.

Spider 2.0-Snow (547 tasks). These are the same 547 questions as Spider 2.0-lite, with every database hosted on Snowflake. Each task asks for one SQL query whose result is compared with the gold result, which is exactly the setting of our method: the generator writes and executes the query, the critic reviews the query and its result, and the forest scores it. We have already run 339 of these questions in their Spider 2.0-lite form (135 SQLite and 204 BigQuery tasks). The remaining 208 need Snowflake. Our pipeline already supports the Snowflake dialect, so completing the set needs no change to the method, only Snowflake access from the Spider 2.0 team. We will add the full 547-task results to the final version, and if access comes through during the discussion period, we will post them here.

Spider 2.0-DBT (68 tasks). This setting replaced the original agentic Spider 2.0 tasks. Each task is a dbt project in which an agent writes several SQL models over many steps, and it is scored by the final state of the database. There is no single query and result for a critic to review, or for the system to accept or abstain on, so per-query confidence and our risk bound do not apply as they stand. Confidence for multi-step agent workflows is a different and open problem, and 68 tasks are also too few to calibrate a risk threshold, which needs a few hundred labelled queries. We will state this scope in the limitations and name it as future work.

On the 339 questions we could run, our logit-free variant is the best estimator, significantly above SIMBA and self-consistency. Adding the remaining Snowflake questions extends the same table with the same protocol rather than changing the method. We hope this clarifies the scope, and we would be glad to provide any further results.

Reviewer_UwE8OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a confidence estimation framework for Text-to-SQL that aims to identify incorrect SQL queries and determine when to abstain from answering. The approach consists of four stages: (1) a generator LLM that produces and executes a SQL query; (2) a critic LLM that validates the query against the original question, database schema, and execution result, including execution-grounded alignment through reverse question reconstruction; (3) extraction of probabilistic and SQL-structural features; and (4) a Random Forest classifier trained on labeled queries to predict correctness. The resulting confidence scores are used with conformal calibration and a risk-controlled abstention rule.

The approach achieves strong results on Spider, outperforming several existing confidence estimation methods, including SAC-Product and Pooled Token-Level.

Critical strengths
  1. Strong empirical results on Spider. The proposed approach achieves substantial improvements over existing confidence estimation methods, particularly in distinguishing correct from incorrect SQL queries.

  2. Comprehensive feature design. The approach combines complementary signals, including generator confidence, critic judgments, execution-grounded alignment, token probabilities, and SQL-structural features. The ablation studies provide useful insight into the contributions of individual components.

  3. Clear problem formulation. The paper provides a clear formulation of confidence estimation and abstention for Text-to-SQL, with a principled approach to conformal calibration and statistical risk control under stated assumptions.

Critical weaknesses
  1. Potentially unfair baseline comparisons. The proposed approach trains a Random Forest classifier on labeled Spider queries, whereas competing methods, such as SAC-Product and Pooled Token-Level, do not appear to receive comparable supervised training. This makes it difficult to determine how much of the improvement comes from the proposed confidence features versus the additional supervised learning. A fairer and more informative comparison would train classifiers on the baseline features using the same labeled data and training protocol.

  2. Limited evaluation and generalization beyond Spider. The main evaluation and comparisons with existing confidence estimation methods are conducted only on Spider. Although the paper reports zero-shot results on BIRD, these are difficult to interpret without comparisons against the same baselines. Furthermore, the risk-controlled threshold calibrated on Spider does not transfer successfully to BIRD, raising concerns about the generalizability of the calibration. Evaluation on more challenging benchmarks, such as Spider 2.0 and BEAVER, would help establish whether the approach remains effective on complex schemas and realistic enterprise-style Text-to-SQL workloads.

  3. Questionable independence of the critic's validation and alignment signals. The critic performs discriminative validation, reverse question reconstruction, and alignment within a single LLM call. However, the original question is included in the same prompt in which the critic is instructed to reconstruct a question independently from the execution result and schema. It is unclear whether the reconstructed question is genuinely independent of the original question or influenced by its presence in the prompt. Moreover, an execution result alone may not uniquely determine the SQL conditions that produced it, limiting the reliability of reverse reconstruction as a validation signal. These issues raise concerns about the independence and effectiveness of the proposed alignment features.

  4. Computational overhead and practical tradeoffs. The proposed approach requires two sequential LLM calls per query, compared with a single generation call for the logit-based baselines, in addition to offline classifier training and calibration. Although the paper reports latency measurements, a more comprehensive accuracy–cost comparison would help determine whether the improvements justify the additional computational complexity, particularly for large-scale deployment.

Questions for authors
  1. Baseline fairness: Can the authors provide a comparison in which baseline confidence features are used to train classifiers, such as Random Forest or logistic regression, on the same labeled training data? This would help isolate the benefits of the proposed features from those of supervised learning.

  2. Generalization: Can the authors compare their approach with existing baselines on BIRD and extend the evaluation to more challenging benchmarks, such as Spider 2.0 and BEAVER? In particular, how can the observed failure of the Spider-calibrated risk threshold to transfer to BIRD be addressed?

  3. Critic independence: How do the authors ensure that reverse question reconstruction is independent of the original question when both are provided within the same critic prompt? Have they considered evaluating reconstruction in a separate LLM call without exposing the original question?

5 comments
AuthorsOpenReview ↗

Response (Part 1/2): Trained baselines (W1, Q1) and critic independence (W3, Q3)

Thank you for the detailed review. We are glad you found the results strong and the feature design comprehensive. Due to the 5,000 character limit, we answer in two parts and group each weakness with its question.

Baseline fairness (W1, Q1). Our forest is trained on labelled queries but the baselines are not. Train them on the same data (Random Forest or logistic regression) to separate the features' benefit from supervision.

The paper's Table 4 compared our trained estimator with untrained baselines, as published. To answer your question, we trained the baselines exactly as we train ours.

Experiment. Same labels, same 60/20/20 splits and 10 seeds, same Random Forest search, and logistic regression as a second learner, so only the features differ. Logistic regression on baseline scores is multivariate Platt scaling, the supervised recalibration Liu et al. (2025) use for Text-to-SQL. Besides Olmo-3-32B (paper's Table 4), we ran two stronger generators. Each row adds one block of features to the row above and retrains the same forest (10-split AUARC):

FeaturesCallsOlmo-3-32BGPT-OSS-120BQwen3.8-27B
Pooled, untrained180.781.489.3
SAC + Pooled, trained182.686.590.6
+ our token and SQL-structural features184.487.691.0
+ generator's verbalized confidence187.189.293.9
+ critic (Ours)288.590.094.4
Logistic regression: SAC + Pooled / Ours82.1 / 87.885.7 / 89.789.9 / 94.0

The gain comes from the features, not the training. Training helps the baselines by 1.3 to 5.1 AUARC (first two rows), so your concern was justified. With the same training, ours stays 5.9, 3.5 and 3.8 AUARC ahead, and logistic regression gives the same order. Since our features include the baseline ones, their posterior ranks at least as well (Neyman-Pearson), and the table shows how much a finite training set recovers. Most of the gain comes from signals the baselines never use: the generator's verbalized confidence adds 1.6 to 2.9 and the critic 0.5 to 1.4. Under Learn-then-Test at a 15% risk budget on GPT-OSS-120B, the trained baselines certify 1/10 splits with 4.0% coverage, against 10/10 and 52.4% for ours (paper's Table 3).

What we will add. The trained rows replace the untrained ones as the main baselines in the paper's Table 4, and the decomposition goes beside the ablation (Section 5.4).

Critic independence (W3, Q3). The critic reconstructs the question in a call that also sees it, and a result may not determine the SQL. Was reconstruction tested in a separate call without the question?

The paper's critic reconstructs the question in the same call that sees it, so leakage is possible. To answer your question, we redid the reconstruction in separate calls that never see the original question qq.

Experiment. Blind: q~\widetilde q from the schema and result only, the paper's frev(s,r)f_{\mathrm{rev}}(s, r). SQL-aware: q~\widetilde q from the schema, generated SQL and result, which targets the non-uniqueness point. In both, Sim(q,q~)\mathrm{Sim}(q, \widetilde q) is scored by a separate call that sees only the two questions. 3,115 of 3,179 Spider queries completed (64 failed on the gateway), and forest AUARC is the 10-split mean on them. Standalone signals use AUROC, which does not depend on accuracy, so subsets compare fairly.

ReconstructionCopies qq verbatimSimilarity alone, AUROCForest AUARC
Shared (paper)11.8%65.690.4
Blind, separate call0.7%57.890.2
SQL-aware, separate call2.1%59.090.4
No similarity featuren/an/a90.1

Leakage is real, but the estimator does not rely on it. The shared reconstruction copies qq in 11.8% of cases, which inflates its standalone signal. Replacing it with an independent one changes forest AUARC by at most 0.2, within seed variation, and removing it costs 0.3, because the critic's verdict is already a separate feature. A separate reconstruction call would add one LLM call, with its latency and token cost, for no gain in AUARC, so we keep the single call and report the leakage openly.

The second point, that a result does not determine its query, also holds. Execution is many-to-one, so a check based on the result alone cannot separate a correct query from a wrong one with the same result. On the 1,882 queries with a table result, giving the reconstruction the SQL raises its standalone AUROC from 56.7 to 62.0, while on the 1,233 single-value or empty results neither version exceeds 59. This is why the verdict reads qq, the SQL and the result together, and reconstruction is only a secondary signal.

What we will add. The leakage and result-shape analyses go in the appendix, with reverse reconstruction described as a secondary signal.

Please see Part 2 for generalization and computational overhead.

AuthorsOpenReview ↗

Response (Part 2/2): Generalization to BIRD, Spider 2.0-lite and BEAVER (W2, Q2) and cost (W4)

Generalization (W2, Q2). Compare baselines beyond Spider (BIRD, Spider 2.0, BEAVER) and address the Spider threshold failing on BIRD.

We evaluated on BIRD, Spider 2.0-lite and BEAVER, and show how to restore the guarantee. Ours is the full 41 features forest, and logit-free keeps only its 5 LLM reported scores (paper's Table 5). Zero-shot forests are trained on Spider, in-domain ones on the target. Cells give zero-shot AUARC, in-domain in brackets. Random and perfect ranking give ρρ and ρ(1+ln⁡(1/ρ))ρ(1+\ln(1/ρ)) for accuracy ρρ.

(a) BIRD, same generator and dialect as Spider. 1,534 dev queries (GPT-OSS-120B), 34.2% correct, in domain on 60/20/20 splits over 10 seeds. We add SIMBA UQ (Bhattacharjya et al., 2025), reported on Spider and BIRD.

MethodCallsBIRD
Random / perfect34.2 / 70.9
SAC-Product / Pooled136.0 / 37.7
Single CoT Verify246.5
Self-consistency1144.7
SIMBA UQ1146.0 (47.1)
SAC + Pooled, trained144.1 (44.7)
Ours (all 41 features)251.5 (54.6)
Ours, logit-free (5 of 41)251.0 (53.7)

Ours is best in both settings, 7.5 AUARC above SIMBA in domain, and its AUROC (78.5) is near Spider's (79.1), so lower AUARC reflects lower accuracy.

(b) Spider 2.0-lite and BEAVER, new dialects and generator. Spider 2.0-lite (Lei et al., 2025) has 547 tasks: 135 SQLite, 205 BigQuery and 207 Snowflake. We ran 135 SQLite and 204 BigQuery tasks (29.5% correct), as one BigQuery dataset is no longer public. Snowflake needs Spider 2.0 team approval, difficult to obtain during the rebuttal period. On BEAVER (Chen et al., 2024, MySQL) the best of four generators answers only 10.8% of 595 questions even with gold tables given, close to the 11% its authors report. In domain uses 5-fold cross-validation, and self-consistency 3 samples on Spider 2.0 (free BigQuery quota).

MethodCallsSpider 2.0-liteBEAVER
Random / perfect29.5 / 65.510.8 / 34.7
SAC-Product / Pooled132.5 / 33.613.7 / 14.1
Single CoT Verify245.215.1
Self-consistency4 / 1138.719.5
SIMBA UQ1139.6 (39.0)17.6 (18.7)
SAC + Pooled, trained130.8 (33.7)11.1 (13.7)
Ours (all 41 features)240.4 (43.0)16.3 (19.0)
Ours, logit-free (5 of 41)247.9 (47.9)18.8 (19.3)

On Spider 2.0-lite the logit-free variant is best in both settings, significantly above SIMBA (+8.2) and self-consistency (+9.0) in a paired bootstrap, and 2.6 above Single CoT Verify, not significant. On BEAVER it reaches AUROC 73.9 zero-shot, ahead of all one and two call methods, but 64 correct queries give 95% intervals of about ±5 AUARC, too wide to rank the top methods. Training data is also scarce, with 80 correct tasks per training fold on Spider 2.0-lite and 51 on BEAVER, against about 1,470 on Spider, so in-domain training adds little. We recommend the full model when the generator and SQL dialect match training (BIRD), and the logit-free variant when either changes or labels are few (Spider 2.0-lite, BEAVER), since token probabilities learned on Spider only partly carry over.

(c) Restoring the guarantee. A Spider certified threshold does not carry over to BIRD (Section 5.6), and under shift no estimator keeps it without target labels (Tibshirani et al., 2019, Barber et al., 2023). We keep the trained forest and recalibrate only the threshold on a few hundred labelled target queries, a number Proposition 2's binomial test gives in advance. At ε=0.5\varepsilon=0.5, 614 BIRD calibration queries certify all 10 splits (27.9% coverage, 37.4% test risk), and refitting the Spider forest's threshold on 600 labels certifies 82 of 100 runs (19.2%, 38.7%), all below ε\varepsilon.

Computational overhead (W4). Two sequential calls and offline training, against one call for logit baselines. Do the gains justify the cost?

We counted LLM tokens per query over all calls and median latency on 50 queries (sampled calls in parallel, Gemma latency from the paper's Appendix Table 21).

MethodCallsTokens / queryLatency (s)AUARC
SAC-Product / Pooled12,0221.878.6 / 81.4
Self-consistency / SIMBA UQ, K=10K=101122,3833.884.0 / 88.6
Ours, Gemma critic (paper)24,9544.390.0
Ours, Qwen3.5-27B critic, no thinking24,2783.190.0

Ours uses about twice the tokens of the logit baselines (4,278 to 4,954 against 2,022) for 8.6 to 11.4 more AUARC, and sampling methods 4.5 to 5.2 times more than ours for lower AUARC. The fastest critic we tested, Qwen3.5-27B without thinking, keeps 90.0 AUARC at 3.1 s, the setting we recommend.

What we will add. All of this goes into Section 5, with Limitation 4 and Proposition 2 updated.

Thank you again for the thorough review, and please let us know if any further results would help.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer UwE8,

Thank you again for your time and constructive comments. In our response, we trained the baselines on the same labels, tested question reconstruction in separate calls, added BIRD, Spider 2.0-lite and BEAVER, and reported token and latency costs. As the discussion period ends soon, we would like to check whether these address your concerns, and we would be happy to clarify anything further. If they do, we would be grateful if you would consider updating your assessment.

Best regards, The Authors

Reviewer_5p4uOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at confidence estimation and abstention in Text-To-SQL generation. Existing Text-to-SQL systems may generate a valid and executable SQL query for the user's question, but when the query is executed on the backend database, the results obtained may not be correct with respect to the user's original question. The paper develops a generator-critic confidence estimation method for Text-to-SQL that first generates the SQL query and subsequently uses a critic model to validate the query and execution result. The critic's results as well as the generation signals are combined into a feature vector that a probabilistic model (random forest) uses to predict whether to abstain or accept the SQL query. The decision threshold is typically calibrated on a small holdout set. The empirical evaluation is carried out on two well known text-to-SQL datasets (Spider and Bird) using a variety of open weights models. The results demonstrate clearly that the proposed approach improves performance compared with existing text-to-SQL approaches.

Critical strengths
  • The paper is well motivated and considers an important practical problem for which it proposes an efficient solution.

  • The quality of the presentation is overall fairly good. The content is well written and organised and therefore it is fairly easy to follow. The technical details are explained well.

  • The experimental evaluation is sound and covers a fairly wide range of models.

Critical weaknesses
  • Using a second LLM to judge the output of the first LLM in order to improve the performance of the first model or to decide whether the first model should abstain is not new. These methods as well as several variations using ensembles of judges (LLMs) have been around for several years and have been shown to improve performance for different tasks, such as question answering, summarisation, etc. One example is this paper https://arxiv.org/pdf/2402.00367. So in my view, the contribution is fairly limited.

  • Similarity based confidence estimation methods such as SIMBAUQ (https://arxiv.org/pdf/2510.13836) were shown to perform pretty well on Text-To_SQL, especially on Spider and Bird datasets. I think these kinds of methods should be included in the experimental evaluation.

Questions for authors

See weaknesses

6 comments
AuthorsOpenReview ↗

Response (Part 1/2): Contribution beyond LLM judges, with a judge ensemble experiment

Thank you for taking the time to read the paper and for your feedback. We are glad you found the problem well motivated and the evaluation sound. Due to the 5,000 character limit, we answer in two parts. This part explains what is unique in the method and how it improves on the state of the art, with a new experiment on where the gain comes from.

W1. Using a second LLM to judge the output of the first LLM in order to improve the performance of the first model or to decide whether the first model should abstain is not new. These methods as well as several variations using ensembles of judges (LLMs) have been around for several years and have been shown to improve performance for different tasks, such as question answering, summarisation, etc. One example is this paper: https://arxiv.org/pdf/2402.00367. So in my view, the contribution is fairly limited.

With respect, the contribution is not the LLM judge. Our core contributions (Section 1):

  1. Certified risk control. A Learn-then-Test threshold guarantees P(Risk≤ε)≥1−δP(\mathrm{Risk}\le\varepsilon)\ge 1-\delta among accepted queries (Prop. 2), and a conformal p-value bounds false abstention (Prop. 1).
  2. Uncertainty from passive signals. The generator's verbalized confidence, token and SQL-structural features, and one critic's check of the execution result are fused by a Random Forest into p(Z=1∣x)p(Z{=}1\mid x).

The judge is one Stage-2 call. Without it, the forest still reaches 87.1 AUARC on Olmo-3-32B (88.5 with it, 82.6 for the best baseline in the same forest, 10-split means).

Experiment 1: is the gain from the judge, or from the method?

GPT-OSS-120B generates and executes one query per Spider question (n=3,179n=3{,}179, 77.3% correct), and all rows score them. Six critics (Gemma-4-31B, Qwen3.8-27B, DeepSeek-V4.1, Nemotron-3-Super/Ultra, GPT-OSS-20B) each review every query separately on the same question, schema, SQL and result. Rows differ only in how reviews become a score:

  • Single judge: its signed confidence.
  • Vote: the share of 3 or 5 judges that accept.
  • Compete: whether a second generator (Qwen3.8) gets the same result.
  • Judges-only forest: our forest on 3 judges' reviews.
  • Ours: our 41-feature forest.

Vote and Compete follow Feng et al. (2024), where other LLMs review an answer (Cooperate) or propose rival ones (Compete) to decide abstention in QA. We score them from verdicts and execution results. Each score runs through Learn-then-Test (ε=0.15\varepsilon=0.15, δ=0.1\delta=0.1) on 10 resampled splits, and AUARC is the 10-split mean. In the last column, as in the paper's Table 3, ours is 10/10 (52.4%): a threshold passes on every split, so accepted queries have at most 15% error at 90% confidence, and it answers 52.4% of test queries (observed error 9.1%). With 0/10 it must abstain on everything.

MethodCallsAUARCCertified (coverage)
Single judge (6 models)279.7 to 88.80 to 9/10 (up to 32.1%)
3 / 5 judges, vote4 / 681.5 / 82.90/10
Compete285.40/10
Judges-only forest490.510/10 (51.5%)
Ours (Gemma critic)290.010/10 (52.4%)
Ours + 3 judges591.710/10 (65.8%)

What this shows.

  • The judge is not the source. The same critic gains 2.3 to 10.3 AUARC inside our pipeline.
  • Voting over judges certifies nothing. Three judges unanimously accept 68.7% of wrong queries. A kk-vote takes k+1k{+}1 values, so correct and wrong queries often tie (P(s+=s−)=0.66P(s^+=s^-)=0.66 for k=3k=3), and ties cap AUROC: AUROC=P(s+>s−)+12P(s+=s−)≤1−12P(s+=s−)=0.67.\text{AUROC}=P(s^+>s^-)+\tfrac12P(s^+=s^-)\le 1-\tfrac12P(s^+=s^-)=0.67.
  • The judges only forest matches ours because it is our method with judges as inputs, at 4 calls and 2.2× the tokens. Ours needs one critic since generator signals cost no extra call, and the two compose (last row).
  • Why fusion works. By Neyman-Pearson, P(Z=1∣x)P(Z{=}1\mid x) ranks at least as well as any function of one judge's score (one coordinate of xx), and the forest estimates it by stacking.
  • At ε=0.10\varepsilon=0.10, more judges help: judges only forest 9/10 (25.5%), ours 0/10, ours with 3 judges 7/10 (30.7%).

What we will add. We will cite Feng et al. in Section 2, add Experiment 1 to Section 5, and state the contribution as certified risk control over fused passive signals.

Where this sits in the literature. Feng et al. (2024) and Wen et al. (2025) study LLM abstention without a certified error rate. In Text-to-SQL, SQLFixAgent and SQLens detect or repair errors, Chen et al. (2025) certify only schema linking, and estimators use one signal family: logits (Ramachandran & Sarawagi, 2024, Entezari Maleki et al., 2026), sub-clause frequencies (Liu et al., 2025) or sampling (Wang et al., 2023). None combines execution-grounded validation, fusion with generator signals, and a finite-sample bound on accepted-query error.

Please see Part 2 for SIMBA UQ (W2) and linked references.

AuthorsOpenReview ↗

Response (Part 2/2): SIMBA UQ comparison on Spider and BIRD

Thank you again. This part answers your second point, on similarity-based uncertainty.

W2. Similarity based confidence estimation methods such as SIMBAUQ (https://arxiv.org/pdf/2510.13836) were shown to perform pretty well on Text-To_SQL, especially on Spider and Bird datasets. I think these kinds of methods should be included in the experimental evaluation.

Our method outperforms every SIMBA UQ variant, by 1.4 AUARC on Spider and 7.5 on BIRD over the strongest, with only 2 LLM calls instead of 11 (78% fewer tokens per query on Spider). We agree this family was missing, and we added SIMBA UQ (Bhattacharjya et al., 2025) on both Spider and BIRD.

Experiment 2: SIMBA UQ against our method.

We use the GPT-OSS-120B queries from Part 1 on Spider (n=3,179n=3{,}179, 77.3% correct), plus BIRD dev (n=1,534n=1{,}534, 34.2% correct). For each question we draw K=10K=10 extra samples at T=0.7T=0.7 with the same prompt. We score the greedy SQL, the query we execute, by its Jaccard similarity over SQL tokens to each sample, and aggregate the 10 similarities in SIMBA's two ways. arith-agg takes their mean, and clf-pairs+gen fits a depth-4 forest on the sorted similarities and the generator's token probability. We also add two variants suited to SQL, agreement of execution results (+ execution match) and self-consistency. Splits and the Learn-then-Test rule are the same as in Part 1. On BIRD, all forests (SIMBA's and ours) are trained on BIRD, unlike the paper's zero-shot setting, and we use ε=0.5\varepsilon=0.5 because 66% of BIRD queries are wrong.

MethodCallsSpider AUARCSpider certified, ε=0.15\varepsilon=0.15BIRD AUARCBIRD certified, ε=0.5\varepsilon=0.5
Self-consistency, K=10K=101184.00/1041.70/10
SIMBA arith-agg1186.01/10 (4.2%)44.70/10
SIMBA clf-pairs+gen1187.32/10 (10.4%)46.80/10
SIMBA + execution match1188.65/10 (20.3%)47.10/10
Ours290.010/10 (52.4%)54.63/10 (3.8%)
Ours + SIMBA/SC1290.610/10 (60.2%)54.95/10 (11.3%)

What this shows.

  • SIMBA is a strong baseline on Spider, as you say. Its best variant is close to ours in AUARC, but it certifies only 5/10 splits (20.3% coverage) against our 10/10 (52.4%).
  • On BIRD the gap widens to 7.5 AUARC and 9.5 AUROC (10-split means 78.5 vs 69.0), and no sampling method certifies any threshold. Ours certifies 3/10, and its 3.8% coverage counts the other 7 splits as 0, since 307 calibration queries give the test little power.
  • The signals compose. Adding the SIMBA and self-consistency scores to our features raises Spider coverage to 60.2% and BIRD certification to 5/10 splits.

Why similarity misses errors in Text-to-SQL. Sample agreement estimates m(x)=Ps′∼p(⋅∣x)[r(s′)=r(s^)],m(x)=P_{s'\sim p(\cdot\mid x)}[r(s')=r(\hat s)], the mass the generator puts on the greedy query's result. Correctness is a different event, Z=1[r(s^)=r(s∗)]Z=\mathbf{1}[r(\hat s)=r(s^\ast)]. When the generator is consistently wrong, m(x)≈1m(x)\approx 1 while Z=0Z=0, and more samples only estimate mm more precisely. SIMBA's SQL similarity measures the same mode. The data show how often this happens:

SpiderBIRD
Wrong queries where all 10 samples agree51.9%39.5%
Correct queries where all 10 samples agree88.3%63.2%
Our AUROC inside this all-agree set74.677.4
  • These ties alone cap self-consistency on Spider at AUROC ≤1−12(0.519⋅0.883)=0.77\le 1-\frac12(0.519\cdot0.883)=0.77, by the tie bound in Part 1. From K=3K=3 to K=10K=10, its AUROC rises only from 64.5 to 70.5.
  • Where self-consistency gives every query the same score, ours still ranks at 74.6 on Spider and 77.4 on BIRD, close to its AUROC on all queries under the same 5-fold protocol (77.6 and 78.3). The critic reads the execution result, and the generator features do not depend on sampling.
  • Consistently wrong queries are 26% of BIRD and 12% of Spider, which explains the wider gap.

What we will add. A table in Section 5 with SIMBA UQ and self-consistency on Spider and BIRD, including call and token counts, and the all-agree analysis.

Across both parts. Used on their own, a single judge, a vote of judges and sample similarity each give a score, but none certifies on all 10 splits at the paper's Spider budget (ε=0.15\varepsilon=0.15). Ours does with two calls, and it composes with both: 91.7 AUARC with judges and 60.2% coverage with SIMBA. We hope these results address your concerns.

References (others are cited in the paper)

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 5p4u,

Thank you again for your time and constructive comments. In our response, we added a comparison with LLM-judge ensembles following Feng et al. (2024) and a full SIMBA UQ evaluation on Spider and BIRD. As the discussion period ends soon, we would like to check whether these address your concerns, and we would be happy to clarify anything further. If they do, we would be grateful if you would consider updating your assessment.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.