acceptodds
Under review as a conference paper at ICLR 2027

Separating Ordering, Calibration, and Audit Evidence in Text-to-SQL Acceptance

Abstract

Text-to-SQL acceptance can improve without changing SQL candidates or the judge body. A SOURCE-trained Qwen residual readout raises disjoint WikiSQL coverage from 2.066% to 3.327% and reduces accepted-error risk from 7.104% to 2.667%. Producer-aware native-prefix controls and matched scalar-spline readouts retain the ordering advantage on the fixed bank. SOURCE-database exclusions preserve fixed-quota gains but reverse CAL-selected correct-mass gains in two of eight conditions. Across 256 paired calibration draws, scalar-plus-residual correct-mass gain over the scalar readout averages 1.264 percentage points and reverses in 50 draws; the residual policy exceeds one-percent risk in 225. Exact acceptance decompositions and without-replacement confidence sequences separate correct-mass gain, risk reduction, and strict dominance. Every-question inspection resolves correct-mass gain earlier; risk reduction and dominance generally require nearly all accepted-union questions in these replays. Calibration-component matching supplies no consistent benefit. Frozen-readout gains on new Mistral answers to exposed questions remain unestablished. Producer-specific adaptation establishes neither incremental ordering gain nor a sufficiently supported one-percent endpoint. These controls support conditional ordering improvement while separating operating-point stability, portability, and claim-specific audit evidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
text-to-SQL, selective prediction, residual readout, risk calibration, confidence sequences, finite-population auditing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RFqqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how to decide whether to accept a generated SQL query. It trains a readout using a frozen judge’s score and hidden states, and separately examines ranking quality, threshold transfer, and the labels needed to validate error-risk claims. Experiments on WikiSQL and BIRD show gains for Qwen, but do not establish consistent transfer across judges or generators, or reliable acceptance at the 1% error target.

Critical strengths

The comparisons with scalar readouts and producer-aware threshold mixtures are useful. They provide evidence that the Qwen readout improves candidate ranking on the fixed evaluation bank beyond what threshold adjustment alone can achieve (Section 4.1)

Critical weaknesses
  1. The presentation is very poor.
  • Many paper-specific terms are used before they are defined. For example, SOURCE and CAL already appear in the abstract and introduction, but are only explained in Section 3. Some basic terms, such as a WikiSQL “component,” are also not clearly defined. The paper needs a substantial rewrite in plain, readable language and the logic.
  • There is not a single citation in the introduction to support the claims.
  • There is a lot of defensive writing. For example, “Neither relative gain nor a favorable point estimate establishes deployment success” (p. 1) and “No intervention alone establishes one-percent certification” (Figure 1).
  • The number of decimal places is inconsistent, e.g., 3.344% coverage versus 2.653370% risk in the same paragraph in Section 4.
  • Overall, it is very difficult to follow the motivation, research question, and problem definition. The paper moves into detailed experimental results before making these basic points clear.
  1. The research question is not meaningful and hard to understand. My understanding is that the paper trains a readout to improve the judgment of SQL generated by text-to-SQL systems. However, I am not convinced that the paper explains why its specific research question is meaningful beyond this narrow setting. The main experiments use fixed candidates from two SQL producer families and two judges. Although an additional Mistral producer is tested, the improvement does not clearly transfer to it (Section 7, Table 3). I do not see sufficient evidence that the findings generalize to broader settings or real-world scenarios with diverse requests.

  2. The research scope is very narrow, only focusing on some small language models. More SOTA models are not evaluated, e.g. GPT series models. All of claims and experiments are established on 2~3 small language models and very outdated benchmarks, e.g. wikisql. These benchmarks already could be handled by current SOTA models. Thus, it is hard to get the value of this work.

  3. Important technical details are unclear or missing. For example, the full judge prompts are not provided, and the construction of the “bounded packet of returned values” used in VALUES is not explained in enough detail. The generation setup for the original SQL candidates and the training data and procedure for the fixed Qwen VALUES adapter are also insufficiently documented. These details are needed to understand and reproduce the experiments, including whether the adapter’s training data were kept separate from the evaluation data.

Questions for authors

See the above section.

Reviewer_WzQuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates selective acceptance in Text-to-SQL systems by separating three aspects of reliability: candidate ordering, calibration transfer, and statistical evidence for acceptance decisions. Using frozen SQL candidates and judge models, the authors evaluate SOURCE-trained residual readouts that combine native confidence scores with hidden-state information. On the component-disjoint WikiSQL evaluation, the Qwen-based residual readout improves coverage from 2.066% to 3.327% while reducing accepted-error risk from 7.104% to 2.667%. The authors introduce stronger producer-aware native-score controls and scalar-spline comparisons to examine whether the improvements can be explained by alternative confidence scoring methods. They also study calibration stability through SOURCE-database exclusions, calibration-donor crossovers, and 256 repeated calibration draws. Finally, they investigate label-efficient auditing using different question-sampling strategies and confidence sequences. Results show that residual ordering improves acceptance in some settings, but the improvements are not consistent across judges, calibration samples, or new SQL producers, and the one-percent error-risk target is not reliably achieved.

Critical strengths
  • The paper addresses an important problem in Text-to-SQL reliability by distinguishing improvements in confidence ranking from calibration quality and statistical guarantees on accepted answers.
  • The authors provide detailed experimental controls, including producer-aware native-score comparisons, scalar-only baselines, SOURCE-database exclusions, and calibration-donor crossovers. These analyses help distinguish genuine ordering improvements from alternative explanations.
  • The evaluation is transparent about negative results and limitations. In particular, the authors report calibration instability, unsuccessful transfer to new SQL producers, and the failure to reliably achieve the one-percent accepted-error target.
  • The statistical analysis is comprehensive, including explicit treatment of unresolved outcomes, question-level sampling, and confidence sequences for different acceptance claims.
Critical weaknesses
  • I am not sure how much methodological novelty this work provides beyond existing selective classification and calibration methods. The paper offers detailed analyses but introduces no new verification model or acceptance algorithm.
  • The practical improvements are limited and inconsistent. The Qwen residual readout still exceeds the 1% error-risk target (2.667%), and the gains do not consistently transfer to Granite or Mistral.
  • Although the authors compare against a five-knot scalar-spline baseline, all fitted scalar functions remain monotonic. Stronger nonlinear scalar-only baselines would help establish the necessity of hidden-state residual features.
  • The auditing procedure requires nearly all accepted-union questions to establish stronger claims such as risk reduction and strict dominance, limiting its practical label efficiency.
Questions for authors
  • Would more flexible nonlinear scalar-only baselines reduce the advantage of the hidden-state residual readout?
  • Under what operating conditions would the proposed approach provide stable improvements over native confidence scores while maintaining sufficient acceptance support?
  • Why do the residual readouts improve Qwen results but fail to show consistent gains with Granite or Mistral? Could the authors elaborate on the factors behind these differences?
Reviewer_x7m8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when a text to SQL system should accept a generated query or abstain, aiming to keep wrong answers below one percent of accepted queries. Holding the candidate queries and the judge model fixed, it separates three questions that are often mixed together. These are whether a new score ranks candidates better, whether a calibrated threshold carries over to new data, and how many labels are needed to audit a fixed policy. On WikiSQL with a Qwen judge, adding a signal from the judge's hidden states raises coverage from about two to about three percent and lowers the error rate among accepted queries from about seven to under three percent. The gain survives strong hindsight controls. The target is still not met with adequate support, matching calibration data to evaluation data does not help, and showing lower risk needs labels for nearly all affected questions.

Critical strengths
  1. I think separating ranking, threshold transfer and audit evidence is a useful lesson for anyone evaluating abstention. The hindsight control that gives each producer its own thresholds on the native score is a demanding test, and the residual gain passes it (Table 1).

  2. I appreciate how candid the paper is. It reports adverse Granite results and the failed transfer to a new producer, and it never presents a favourable point estimate as a guarantee.

Critical weaknesses
  1. The headline operating point seems fragile. In the repeated calibration analysis the residual readout exceeds one percent risk in 225 of 256 draws, meets both the risk target and the support screen in only one, and has a mean coverage near 1.8 percent rather than 3.3 percent (Table 13).

  2. The positive result is narrow. It holds for one judge on one main dataset, every Granite contrast across repeated calibration draws is negative, and the readout is fitted on only 857 candidates from eight databases, where removing either of two databases reverses the calibrated gain.

  3. Coverage stays near three percent and the target is never reached with enough accepted questions, so I feel the practical value is not yet shown. The audits also save few labels for the risk claims, which are first supported after about 96 percent of the affected questions are labelled.

  4. I found the paper very hard to read. Dense acronyms and heavily hedged sentences hide the main message, and the first appendix has a heading but no content.

Questions for authors
  1. How was the Qwen adapter selected, and did any of the previously exposed banks inform that choice?

  2. How sensitive is the gain to the regularization strength and to using only the final layer, both of which appear fixed throughout?

  3. About a quarter of the new producer outcomes on WikiSQL are unresolved. How much do the conclusions there depend on how these are completed?

Suggestions

  1. Reporting the main ranking contrast with a larger fitting set or on a second dataset might show whether the gain holds beyond this setup.

  2. An early plain table of what improves and what does not might help readers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.