- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents EviSQL, a reference-free framework for verifying whether a candidate SQL query satisfies the semantics of a natural-language question. The framework searches for execution evidence both within the candidate query and beyond its computation: it reconstructs a computation trace and tests selected operations through validated local perturbations, then audits question-grounded requirements and uses targeted exploratory SQL to investigate possible coverage gaps. A final-impact check determines whether observed discrepancies affect the answer, and the resulting evidence is used for error diagnosis and SQL correction. The authors evaluate EviSQL on ROSE-VEC and BIRD-VC for semantic error detection and diagnosis-guided correction, and on BIRD-CRITIC-SQLite for resolving user-reported SQL issues. They report improvements over the evaluated baselines in error-detection metrics, correction accuracy, and issue-resolution success rate, along with ablation and cost analyses.
Critical strengths
-
Comprehensive verification framework. EviSQL combines in-trace verification, out-of-trace exploration, and final-impact judgment to examine both the computations performed by a candidate SQL query and potentially omitted requirements from the natural-language question. This provides a systematic approach to semantic verification beyond execution-based checking alone.
-
Promising empirical performance. EviSQL is evaluated on multiple datasets and compared with several relevant baselines. It achieves the best reported Cohen’s Kappa and F1 scores on both ROSE-VEC and BIRD-VC, and its diagnosis-guided correction experiments show further improvements in verified accuracy.
Critical weaknesses
-
Uncertainty in intent extraction and mapping. The out-of-trace exploration component relies on extracting requirements from natural-language questions and mapping them to computation traces. However, the paper acknowledges that its consistency checks do not guarantee complete intent extraction or semantically valid mappings. More direct evaluation of intent coverage and mapping correctness would strengthen the reliability claims.
-
Potential limitations in evaluation-label reliability and generalization. BIRD-VC uses corrected SQL queries as evaluation references, raising questions about how the correctness of these references is independently established. In addition, further evaluation on diverse and challenging semantic-error cases would help determine whether the reported gains generalize beyond the current benchmarks.
Questions for authors
-
Intent extraction and trace alignment.
What evidence do the authors have that the intent inventory is sufficiently complete and that natural-language requirements are mapped correctly to the computation trace? In particular, can the authors report an evaluation of intent coverage and mapping correctness, or clarify how the framework avoids false verification when a requirement is omitted or incorrectly aligned?
-
Reliability of BIRD-VC labels.
How were the corrected SQL queries used as reference labels in BIRD-VC independently validated? Were they reviewed by human annotators or checked through another procedure, and how were disagreements or cases where multiple SQL queries satisfy the question handled?
-
Cost-effectiveness.
Given the additional LLM calls and token consumption reported for EviSQL, how should its accuracy gains be weighed against its inference cost in practical use? Can the authors provide or discuss a cost-quality comparison (for example, against the ablated variants or baselines) that identifies settings where the added cost is justified?
Reliability and Cost of Intent-Based Verification
Q1: Intent extraction and trace alignment.
We appreciate this concern about the reliability of intermediate judgments. Intent extraction and semantic alignment rely on the LLM's interpretation of the question. Each extracted Intent is mapped to the computation trace or investigated as a potential missing requirement. Execution records and proposed local edits undergo programmatic validation. Before returning a completed \textsc{Correct} verdict, all extracted Intents must be reviewed and final synthesis must find no supported semantic conflict. \textsc{Checked} records completion of an investigation. Sec. 4.4 and Appendix Sec. A.3 describe the final-decision procedure. Appendix Sec. A.4 illustrates how requirements and trace mappings connect to execution evidence and the verdict.
We assess diagnosis quality in Sec. 5.5 by matching diagnoses to human error descriptions with a fixed LLM judge, obtaining the highest HEC of 80.32%. Sec. 5.3 uses the same LLM fixer and repair prompt for queries flagged as incorrect, with each method's diagnoses as guidance. Over all 351 candidates, EviSQL achieves 92.02% human-verified post-correction accuracy, compared with 72.93% before repair. In additional results reported here, five ROSE-VEC runs give a sample standard deviation of 0.2545 in 100κ. Three LLM-generated prompt rewrites vary by at most 1.2 points, all scoring above the compared baselines. These results evaluate the resulting diagnoses and decisions. Independent human assessment of Intent coverage and mapping correctness remains pending.
Q2: Reliability of BIRD-VC labels.
Our annotation references come from the publicly released Arcwise-Plat-SQL resource \citep{jin2026pervasive}. The source study reports that two authors independently assessed cases flagged as incorrect or ambiguous, with disagreements discussed by all four authors. Following the human semantic-annotation approach of NL2SQL-BUGs, we collected and annotated candidates from six generators, retaining each candidate's generating method. Three researchers working on LLMs and databases annotated disjoint subsets, with one initial annotation per candidate. A fourth researcher with extensive Text-to-SQL experience reviewed the annotations and adjudicated uncertain cases and disagreements arising during review.
Annotators judge each candidate against the question, evidence, and schema, using the corrected SQL as a reference and accepting alternative implementations satisfying those requirements. Representation is assessed in context: 0.25 and 25% denote the same proportion, while any requested output scale must be followed. Explicit requirements are checked even when execution results match the reference. Appendix Sec. A.5 describes our candidate-annotation procedure and ambiguity handling. The source-study review details above are provided here.
Q3: Cost-effectiveness.
We acknowledge that multi-round verification adds inference cost. Text-to-SQL generation methods such as MAC-SQL and CHASE-SQL also use multi-stage inference. EviSQL uses additional rounds to acquire and examine verification evidence. The table below combines the ablation results and token costs reported in Sec. 5.6. The full-model reference uses Table 1's penalized scoring (Appendix Sec. A.3).
Removing out-of-trace exploration saves 36.1%/35.4% of tokens while losing 4.31/2.56 points in 100κ on ROSE-VEC/BIRD-VC. Final-impact judgment improves 100κ by 0.18/1.19 points while reducing token consumption by 15--16%. A broader comparison of baseline costs and end-to-end latency remains to be completed.
Additional response to Weakness 2: Generalization.
The cross-generator comparison in Sec. 5.2 covers six generators. EviSQL's penalized scores exceed the strongest compared baseline for each generator by 1.60--11.04 points in 100κ. ROSE-VEC includes candidates originating from both BIRD and Spider. New Sec. 5.7 analyzes all 175 incorrect ROSE-VEC queries, with descriptive grouping criteria in Appendix Sec. A.5. EviSQL detects 163/175 errors overall, including 37/43 schema-or-join errors (86.05%) and 30/34 aggregation-or-granularity errors (88.24%). These two groups account for 10 of the 12 undetected errors, suggesting that entity alignment and aggregation scope deserve further study.