acceptodds
Under review as a conference paper at ICLR 2027

Within and Beyond SQL: Searching for Execution Evidence in Text-to-SQL Verification

Abstract

Text-to-SQL maps natural language questions to executable SQL queries over databases. Reliable text-to-SQL requires verifying whether generated SQL faithfully satisfies the semantics of the question. Existing verification remains challenging because intermediate discrepancies may be masked by downstream operations, while omitted requirements leave no corresponding computation from which execution evidence can be directly obtained. We introduce EviSQL, a reference-free verification framework that searches for execution evidence both within and beyond the computation expressed by candidate SQL. Within the SQL, EviSQL reconstructs the computation trace of SQL and tests selected operations through local perturbations, tracking whether the resulting differences propagate to the final answer. Beyond the SQL, EviSQL grounds question requirements in the original wording and relates them to the computation trace, using intent-bound queries to investigate possible semantic coverage gaps. Experiments on \rose and \bird show that EviSQL consistently improves semantic error detection over existing verification methods. The resulting execution evidence further supports fine-grained error diagnosis and targeted SQL correction, improving the reliability of generated queries. Code and experimental resources are available at https://anonymous.4open.science/r/EviSQL-30D4.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Text-to-SQL, Semantic Verification, Execution Evidence

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Nwh6OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents EviSQL, a reference-free framework for verifying whether a candidate SQL query satisfies the semantics of a natural-language question. The framework searches for execution evidence both within the candidate query and beyond its computation: it reconstructs a computation trace and tests selected operations through validated local perturbations, then audits question-grounded requirements and uses targeted exploratory SQL to investigate possible coverage gaps. A final-impact check determines whether observed discrepancies affect the answer, and the resulting evidence is used for error diagnosis and SQL correction. The authors evaluate EviSQL on ROSE-VEC and BIRD-VC for semantic error detection and diagnosis-guided correction, and on BIRD-CRITIC-SQLite for resolving user-reported SQL issues. They report improvements over the evaluated baselines in error-detection metrics, correction accuracy, and issue-resolution success rate, along with ablation and cost analyses.

Critical strengths
  1. Comprehensive verification framework. EviSQL combines in-trace verification, out-of-trace exploration, and final-impact judgment to examine both the computations performed by a candidate SQL query and potentially omitted requirements from the natural-language question. This provides a systematic approach to semantic verification beyond execution-based checking alone.

  2. Promising empirical performance. EviSQL is evaluated on multiple datasets and compared with several relevant baselines. It achieves the best reported Cohen’s Kappa and F1 scores on both ROSE-VEC and BIRD-VC, and its diagnosis-guided correction experiments show further improvements in verified accuracy.

Critical weaknesses
  1. Uncertainty in intent extraction and mapping. The out-of-trace exploration component relies on extracting requirements from natural-language questions and mapping them to computation traces. However, the paper acknowledges that its consistency checks do not guarantee complete intent extraction or semantically valid mappings. More direct evaluation of intent coverage and mapping correctness would strengthen the reliability claims.

  2. Potential limitations in evaluation-label reliability and generalization. BIRD-VC uses corrected SQL queries as evaluation references, raising questions about how the correctness of these references is independently established. In addition, further evaluation on diverse and challenging semantic-error cases would help determine whether the reported gains generalize beyond the current benchmarks.

Questions for authors
  1. Intent extraction and trace alignment.

    What evidence do the authors have that the intent inventory is sufficiently complete and that natural-language requirements are mapped correctly to the computation trace? In particular, can the authors report an evaluation of intent coverage and mapping correctness, or clarify how the framework avoids false verification when a requirement is omitted or incorrectly aligned?

  2. Reliability of BIRD-VC labels.

    How were the corrected SQL queries used as reference labels in BIRD-VC independently validated? Were they reviewed by human annotators or checked through another procedure, and how were disagreements or cases where multiple SQL queries satisfy the question handled?

  3. Cost-effectiveness.

    Given the additional LLM calls and token consumption reported for EviSQL, how should its accuracy gains be weighed against its inference cost in practical use? Can the authors provide or discuss a cost-quality comparison (for example, against the ablated variants or baselines) that identifies settings where the added cost is justified?

1 comment
AuthorsOpenReview ↗

Reliability and Cost of Intent-Based Verification

Q1: Intent extraction and trace alignment.

We appreciate this concern about the reliability of intermediate judgments. Intent extraction and semantic alignment rely on the LLM's interpretation of the question. Each extracted Intent is mapped to the computation trace or investigated as a potential missing requirement. Execution records and proposed local edits undergo programmatic validation. Before returning a completed \textsc{Correct} verdict, all extracted Intents must be reviewed and final synthesis must find no supported semantic conflict. \textsc{Checked} records completion of an investigation. Sec. 4.4 and Appendix Sec. A.3 describe the final-decision procedure. Appendix Sec. A.4 illustrates how requirements and trace mappings connect to execution evidence and the verdict.

We assess diagnosis quality in Sec. 5.5 by matching diagnoses to human error descriptions with a fixed LLM judge, obtaining the highest HEC of 80.32%. Sec. 5.3 uses the same LLM fixer and repair prompt for queries flagged as incorrect, with each method's diagnoses as guidance. Over all 351 candidates, EviSQL achieves 92.02% human-verified post-correction accuracy, compared with 72.93% before repair. In additional results reported here, five ROSE-VEC runs give a sample standard deviation of 0.2545 in 100κ100\kappa. Three LLM-generated prompt rewrites vary by at most 1.2 points, all scoring above the compared baselines. These results evaluate the resulting diagnoses and decisions. Independent human assessment of Intent coverage and mapping correctness remains pending.

Q2: Reliability of BIRD-VC labels.

Our annotation references come from the publicly released Arcwise-Plat-SQL resource \citep{jin2026pervasive}. The source study reports that two authors independently assessed cases flagged as incorrect or ambiguous, with disagreements discussed by all four authors. Following the human semantic-annotation approach of NL2SQL-BUGs, we collected and annotated candidates from six generators, retaining each candidate's generating method. Three researchers working on LLMs and databases annotated disjoint subsets, with one initial annotation per candidate. A fourth researcher with extensive Text-to-SQL experience reviewed the annotations and adjudicated uncertain cases and disagreements arising during review.

Annotators judge each candidate against the question, evidence, and schema, using the corrected SQL as a reference and accepting alternative implementations satisfying those requirements. Representation is assessed in context: 0.250.25 and 25%25\% denote the same proportion, while any requested output scale must be followed. Explicit requirements are checked even when execution results match the reference. Appendix Sec. A.5 describes our candidate-annotation procedure and ambiguity handling. The source-study review details above are provided here.

Q3: Cost-effectiveness.

We acknowledge that multi-round verification adds inference cost. Text-to-SQL generation methods such as MAC-SQL and CHASE-SQL also use multi-stage inference. EviSQL uses additional rounds to acquire and examine verification evidence. The table below combines the ablation results and token costs reported in Sec. 5.6. The full-model reference uses Table 1's penalized scoring (Appendix Sec. A.3).

MethodROSE-VEC 100κ100\kappaROSE-VEC TokensBIRD-VC 100κ100\kappaBIRD-VC Tokens
Full (penalized)70.4619,73882.4820,622
No in-trace67.9817,92679.8218,860
No out-of-trace66.1512,60379.9213,325
No final-impact70.2823,33881.2924,659

Removing out-of-trace exploration saves 36.1%/35.4% of tokens while losing 4.31/2.56 points in 100κ100\kappa on ROSE-VEC/BIRD-VC. Final-impact judgment improves 100κ100\kappa by 0.18/1.19 points while reducing token consumption by 15--16%. A broader comparison of baseline costs and end-to-end latency remains to be completed.

Additional response to Weakness 2: Generalization.

The cross-generator comparison in Sec. 5.2 covers six generators. EviSQL's penalized scores exceed the strongest compared baseline for each generator by 1.60--11.04 points in 100κ100\kappa. ROSE-VEC includes candidates originating from both BIRD and Spider. New Sec. 5.7 analyzes all 175 incorrect ROSE-VEC queries, with descriptive grouping criteria in Appendix Sec. A.5. EviSQL detects 163/175 errors overall, including 37/43 schema-or-join errors (86.05%) and 30/34 aggregation-or-granularity errors (88.24%). These two groups account for 10 of the 12 undetected errors, suggesting that entity alignment and aggregation scope deserve further study.

Reviewer_snciOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EviSQL, a reference-free framework for Text-to-SQL semantic verification. It constructs SQL computation traces and combines local SQL perturbations with intent-based exploration to detect incorrect operations and missing requirements. Execution evidence is used to support error detection, diagnosis, and correction. Experiments on ROSE-VEC, BIRD-VC, and BIRD-CRITIC-SQLite demonstrate improvements over existing verification and repair methods.

Critical strengths

Important Problem: The paper addresses reference-free semantic verification for Text-to-SQL, which is an important and practical problem.

Comprehensive Framework: EviSQL considers both errors within SQL computations and missing requirements, providing a systematic verification framework.

Promising Results: The method achieves strong performance on multiple benchmarks, demonstrating improvements in SQL error detection and correction.

Critical weaknesses
  1. High Verification Cost: EviSQL requires multiple LLM calls and database executions, consuming approximately 20K tokens per sample. Such overhead raises concerns about its practicality and scalability.

  2. Questionable Value of Local SQL Perturbation: Modifying a local SQL operation may naturally produce different execution results, even when the original SQL is correct. Conversely, semantically different SQL queries may produce identical results, particularly on empty or sparse databases. Therefore, the additional information provided by local perturbation is unclear, and its necessity is not sufficiently justified.

  3. Heavy Dependence on LLM Judgements: Key steps, including suspicious node selection, local SQL modification, and semantic interpretation, rely heavily on the LLM. Programmatic execution checks cannot guarantee the correctness of these semantic judgements.

  4. Insufficient Robustness Analysis: The experiments lack detailed evaluations of cases where execution results cannot distinguish semantically different SQL queries, as well as performance breakdowns across different SQL error types.

  5. The related work section overlooks representative studies on fine-tuning-based Text-to-SQL error detection, such as Error Detection for Text-to-SQL Semantic Parsing (Findings of EMNLP 2023) and SQL-Checker: Error Detection and Labeling for Text-to-SQL with Interpretability Analysis (WWW 2026). Additionally, the evaluation does not include the NL2SQL-Bugs dataset, limiting the comprehensiveness of the experimental comparison.

Questions for authors

What is the purpose of comparing execution results before and after local SQL perturbations? Such modifications may naturally produce different results even when the original SQL is correct, while semantically different SQL queries may produce identical results on a given database instance. How does this comparison provide meaningful additional evidence for determining the semantic correctness of the original SQL?

1 comment
AuthorsOpenReview ↗

Grounding Local Perturbations in Question Semantics

Q1. Purpose of local execution comparisons.

We agree that different execution results alone do not establish a semantic error, and matching results do not establish correctness. A local perturbation tests a hypothesis grounded in the question. We check whether its effect reaches the final answer and violates the requested semantics. The perturbed SQL need not be a complete correction. When current data mask a difference, final review also examines conflicts between the question and the full computation. Secs. 4.2 and 4.4 describe the local-edit and final-answer checks, with execution contracts in Appendix Sec. A.2.

Additional W1. Verification cost.

We recognize that multi-round verification limits applicability when cost or latency is a priority. Text-to-SQL generation methods such as MAC-SQL and CHASE-SQL also use multi-stage inference. EviSQL uses additional rounds to acquire and examine verification evidence. The value of this extra inference still requires a quality and cost comparison. Sec. 5.6 reports the component costs and ablation results for this discussion.

Additional W2. Value of local perturbations.

In addition to Q1, perturbations allow a model's concern to be tested through execution. For a question asking for employee counts, comparing \texttt{COUNT(*)} with \texttt{COUNT(DISTINCT employee_id)} can reveal duplicate counting after a join. The requested counting population provides the semantic basis for this check. Sec. 5.5 compares diagnoses with human error descriptions, obtaining an HEC of 80.32%. Sec. 5.3 supplies each method's diagnoses to the same LLM fixer and repair prompt, with EviSQL obtaining 92.02% human-verified accuracy. These results support diagnosis quality and repair utility, while leaving independent validation of intermediate evidence open. The evaluation settings are described in Appendix Sec. A.6.

Additional W3. LLM judgments.

We agree that question interpretation, suspicious-node selection, and semantic explanation remain dependent on the LLM. Programmatic checks constrain the proposed edits and verify execution provenance and final-answer effects. They do not certify the semantic interpretation. The HEC and human-verified repair results provide indirect evidence of diagnosis utility. Appendix Sec. A.3 states the limitations of quotation checks and model interpretation.

Additional W4. Robustness and error types.

We have added Sec. 5.7 to analyze all 175 incorrect ROSE-VEC queries by their primary semantic defect. The grouping uses the question, SQL, and available human error descriptions, with criteria in Appendix Sec. A.5. The Queries column gives the number of incorrect candidates of each type. Recall is the percentage detected. Unfinished incorrect queries count as misses and remain in the denominator.

Error typeQueriesRecall (%)
Predicates/logic5698.21
Schema/joins4386.05
Aggregation/grain3488.24
Calculation/format13100.00
NULL handling12100.00
Ordering/top-kk7100.00
Output requirements8100.00
Unclassified250.00
All incorrect queries17593.14

Schema/join and aggregation/granularity cases account for 10 of the 12 missed errors, including three unfinished runs. A dedicated evaluation of execution-indistinguishable cases remains to be completed. Sec. 4.4 notes that current data can mask a violation of the question.

Additional W5. Related work and evaluation coverage.

Thank you for pointing out these omissions. We have expanded the discussion of \textit{Error Detection for Text-to-SQL Semantic Parsing} and \textit{SQL-Checker} in Appendix Sec. A.6. The former uses CodeBERT and graph neural networks and provides code, preprocessed data, and checkpoint links. A comparison with its original trained model remains outside our common generative-LLM evaluation. We have not been able to obtain the training data needed to reproduce SQL-Checker.

BIRD-VC follows the human semantic-annotation approach of NL2SQL-BUGs. The public instances identify databases but do not provide candidate-to-generator identifiers for the analyses in Tables 2 and 3. We therefore collect and annotate predictions from six generators while retaining their provenance. Appendix Sec. A.5 describes the annotation approach and retained generator provenance. Evaluation on NL2SQL-BUGs would offer a useful additional comparison and remains a limitation of the current study.

Reviewer_rHh2OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed EviSQL, a reference-free verification framework that searches for execution evidence both within and beyond the candidate SQL. For Within-SQL verification, it reconstructs the trajectory, and checks whether the perturbations affects the final answer. For Beyond-SQL verification, it investigates the gaps between the trajectory and intents. The reported results demonstrates its effectiveness on several datasets.

Critical strengths

S1. The final answer check after local perturbation avoids the false-negative case that the answer-irrelevant perturbations is different with original trajectory.

S2. The verification is seperated into within/beyond-SQL, which points out the important requirement that locally-wrong case and insufficient coverage should be analyzed seperately.

Critical weaknesses

W1.For in-trace verification, the programmatic checks establish only that a proposed perturbation is syntactically localized; they do not establish that the perturbation represents a semantically valid counterfactual for the question. In §4.2, the paper explicitly states that local validation “establishes the scope of the edit rather than its semantic correctness” (p. 5), and requires the model to explain why the selected operation conflicts with the question. The same dependence appears in §4.4: Eq. (5) requires the witness to satisfy Groundedθ(w; Q), while the paper states that the model must explain why the observed discrepancy contradicts QQ (p. 6). However, the paper provides no independent evaluation of the accuracy or coverage of model-generated perturbations, Intent-to-trace mappings, or answer-impact witnesses. Consequently, the reported end-task improvements do not establish that the evidence produced by EviSQL is itself semantically valid or reliable.

W2. Algorithm 4 is not fully consistent with the uncertainty states introduced in out-of-trace exploration. Section 4.3 states that the program checks quotation and identifier consistency “without establishing the completeness of II or the semantic validity of the mappings” (p. 5), and further clarifies that a CHECKED Intent does not imply correct implementation. Nevertheless, Algorithm 4 returns CORRECT whenever no witness or grounded semantic conflict is found, without providing an explicit INCOMPLETE or abstention outcome. Thus, missing Intents or insufficient exploration can be converted into a correctness verdict: an incorrect SQL query may be accepted as correct, corresponding to a false negative for error detection (or a false positive for correctness prediction). The issue is compounded by the fact that Algorithm 4 returns INCORRECT when required reviews are incomplete, whereas Algorithm 3 returns INCOMPLETE when no admissible exploratory SQL remains. The paper should clarify the intended semantics of these states and quantify the resulting false-negative and false-positive rates.

Questions for authors

Q1. How do you verify that a model-proposed perturbation is semantically valid for the original question, rather than merely a syntactically localized edit? Please provide an independent human evaluation or annotated accuracy measure.

Q2.How sensitive are the generated perturbations, Intent mappings, and final decisions to LLM sampling, prompt variations, or repeated runs?

Q3. If the Intent inventory is incomplete or an Intent is only CHECKED, under what conditions can Algorithm 4 safely return CORRECT?

Q4. How does EviSQL distinguish “no semantic error was found” from “the exploration failed to find sufficient evidence”? Please report the false-negative rate caused by incomplete Intent extraction or insufficient exploration.

Q5. Why does Algorithm 4 return INCORRECT when required reviews are incomplete, whereas Algorithm 3 returns INCOMPLETE when no admissible exploratory SQL remains? Are these intended to represent different states?

1 comment
AuthorsOpenReview ↗

Clarifying Diagnosis Validity and the Treatment of Incomplete Runs

Q1. Semantic validity and usefulness of diagnoses.

We agree that end-task gains alone do not establish the semantic validity of each perturbation or witness. Sec. 5.5 compares diagnoses with human-annotated error descriptions in ROSE-VEC using a fixed LLM judge to match atomic claims. EviSQL obtains an HEC of 80.32%, the highest among the compared methods. Sec. 5.3 uses the same LLM fixer and repair prompt for all methods on 351 DeepEye-SQL candidates. Only queries predicted as incorrect are repaired, with each method's diagnoses supplied as guidance.

Diagnosis sourceEX (%)VA (%)
Original, no repair73.2272.93
SAR-Agent87.7590.88
EviSQL90.3192.02

VA is human-verified. EX/VA improve by 17.09/19.09 points, supporting diagnosis-guided repair. EX/VA are measured over all 351 candidates. Each diagnosis must explain which question requirement is violated and how it affects the answer. A perturbation tests a local hypothesis (Sec. 4.2 and Appendix Sec. A.2). Independent human evaluation of perturbations and mappings remains a limitation.

Q2. Sampling, prompt variation, and repeated runs.

We acknowledge that prompt wording and LLM sampling can affect performance. In additional results reported here, three LLM-generated prompt rewrites on ROSE-VEC varied by at most 1.2 points in 100κ100\kappa (0.012 in κ\kappa). All three scored above the compared baselines. Across five ROSE-VEC inference runs, sample standard deviations (denominator 5−15-1) were s100κ=0.2545s_{100\kappa}=0.2545 and sκ=0.002545s_\kappa=0.002545. BIRD-VC repeated-run results remain pending.

Q3. What do CHECKED and Correct mean?

CHECKED records a completed intent-bound investigation. It does not itself set the candidate verdict. Final synthesis may still reject a surviving semantic conflict. A completed candidate is labeled Correct when the audits and final synthesis find no supported semantic conflict under the available evidence and adopted question interpretation. Appendix Sec. A.3 describes this conditional judgment. Sec. 4.4 requires completed reviews before semantic synthesis.

Q4. No error found versus insufficient evidence.

The table reports counts of explicit wrong verdicts and unfinished runs separately.

OutcomeROSEBIRD
Explicit false acceptance990
Explicit false rejection6680
Unfinished, truly incorrect32
Unfinished, truly correct16
All unfinished48
All candidates5852,106

Unfinished runs comprise 0.68% and 0.38% of the respective datasets. The 12 unfinished runs include six final-result cap failures, two intermediate-completeness failures, two Intent-text violations, one witness-arity violation, and one uncovered-Intent completion attempt. The six cap failures concern one question across generators: 29,936 rows exceed the 20,000-row bound. None produced a completed Correct verdict. Appendix Sec. A.3 reports completion and verdict counts. The failure-cause breakdown above is provided here. We cannot yet attribute false acceptances to specific causes without independent annotation.

Q5. Why INCORRECT versus INCOMPLETE?

All 12 unfinished runs retain status=incomplete and verdict=null. Table 1 retains penalized scoring: every unfinished run counts as an evaluation error, contributing FN if truly incorrect and FP if truly correct. This penalty changes metric counts without assigning a completed verdict. For comparison, operational scoring flags only completed Incorrect verdicts, treating unfinished runs as unflagged (FN if truly incorrect, TN if truly correct). The following comparison uses the same metric columns as Table 1:

ScoringROSE-VEC 100κ100\kappaROSE-VEC PROSE-VEC AROSE-VEC RROSE-VEC F1F_1BIRD-VC 100κ100\kappaBIRD-VC PBIRD-VC ABIRD-VC RBIRD-VC F1F_1
Penalized70.4670.8786.5093.1480.4982.4889.9291.5589.2989.60
Operational70.7971.1886.6793.1480.6983.0590.5591.8389.2989.92

P/A/R denote precision/accuracy/recall. Classification metrics are percentages. EviSQL remains best in κ\kappa and F1F_1 under either convention. A separate candidate-processing failure policy can return completed Incorrect after final LLM synthesis. None of these 12 runs uses it. Algorithm 4 in Sec. 4.4 separates status from verdict. Appendix Sec. A.3 defines the scoring rules and reports both conventions.

Additional W1. Diagnosis evaluation.

The evidence discussed in Q1 evaluates diagnosis quality and repair utility (Secs. 5.3 and 5.5 and Appendix Sec. A.6). It provides indirect support and leaves independent validation of intermediate evidence open. We discuss W2 in Q3 to Q5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.