acceptodds
Under review as a conference paper at ICLR 2027

Advancing the Text-to-SQL Frontier through Graph-Language Verification

Abstract

Translating natural language queries into Structured Query Language (SQL) in real world databases requires reconciling ambiguous user intent with complex relational schemas. While modern LLMs can generate candidate SQL queries with ease, identifying whether a query is functionally correct remains a fundamental challenge. Existing selection heuristics (e.g. execution-based self-consistency and LLM-as-a-judge evaluators) fail to handle the inherent syntax-semantics asymmetry of SQL. In other words, SQL has many ways to express equivalent queries, but small (single-token) variations can dramatically change a query's result. In this work, we formulate SQL candidate verification as a cross-modal alignment problem between unstructured textual context (user intent, schema definitions, and domain evidence) and structured relational computation graphs. We introduce Warbler, a unified framework coupling execution-guided candidate generation with a learned graph-language verifier which algorithmically compiles SQL into directed acyclic graphs (DAGs), normalizes syntactic variance, and encodes relational inductive biases via a graph-language architecture. On the highly competitive BIRD benchmark, Warbler establishes a new state of the art on the official single model track with 80.04% execution accuracy on the held-out test set, outperforming prior leading method by 2.90 points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Text-to-SQL, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_sczUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents WARBLER, a Text-to-SQL framework combining execution-guided candidate generation and repair with a learned graph-language verifier. Candidate SQL queries are encoded into typed directed acyclic graphs, encoded using a graph neural network, and combined with contextual language representations to predict execution correctness. The highest-scoring candidate is selected as the final prediction. The paper reports 80.04% execution accuracy on the BIRD test set and development-set improvements over execution self-consistency and text-only LLM scoring.

Critical strengths
  1. The paper reports 80.04% execution accuracy on the BIRD held-out test set, outperforming the strongest listed baseline by 2.90 percentage points.
  2. Representing SQL as typed relational graphs makes operators, entities, and their dependencies explicit. This provides a well-motivated structural inductive bias for interpreting SQL and assessing its compatibility with the natural-language question.
Critical weaknesses
  1. A central argument is that semantically equivalent SQL queries are compiled into isomorphic graphs, leading to invariant representations and scores. However, Section 3.2 mainly describes column-to-table resolution, alias standardization, and node deduplication. It does not clearly define the supported rewrite classes or explain how structurally different but semantically equivalent implementations are canonicalized. For example, an EXISTS query and a JOIN followed by DISTINCT can express the same query under suitable constraints while containing different operators and dependencies.GNN permutation invariance does not establish invariance across these alternative SQL implementations; the latter depends on the compiler producing compatible representations. Conversely, queries that happen to return the same result on one database need not be semantically equivalent and should not necessarily be collapsed.

  2. WARBLER is a pointwise learned verifier used to select among generated candidates. Closely related methods already investigate learned SQL selection. CHASE-SQL (ICLR 2025) combines candidate generation and repair with a trained pairwise selector, while GradeSQL trains outcome reward models using execution-based correctness labels to score and rank SQL candidates. These approaches are not discussed or directly compared in the submission.The graph representation could still provide a meaningful contribution, but the current ablations do not isolate that contribution. The text-only baseline in Section 4.3 is described as prompting an LLM to score SQL, whereas WARBLER receives supervised training. Thus, this comparison changes both the representation and the training procedure. Removing the parser and GNN together also leaves the effects of normalization and graph encoding entangled.

Questions for authors
  1. What SQL rewrite classes and database assumptions are covered by the claimed canonicalization guarantee? How does the compiler handle structurally different but equivalent implementations, such as EXISTS versus JOIN with DISTINCT? Please clarify whether the guarantee is restricted to a defined subset of rewrites.

  2. Provide controlled evaluations of equivalent rewrites and meaning-changing perturbations, measuring both score consistency and stability of the final candidate selection. Is “alignment with user intent” operationally defined entirely through reference execution agreement, or is there additional evidence supporting a broader interpretation?

  3. Compare the proposed verifier with trained text-based selectors using the same candidate pool and correctness labels, including a normalized-SQL-text baseline. Please also clarify the specific contribution relative to existing learned selection methods such as CHASE-SQL and GradeSQL.

4 comments

Thank you for your careful review and for asking us to clarify the scope of canonicalization and the verifier comparisons.

Weakness 1 and Question 1 — Scope of canonicalization

We want to clarify that the nice structural equivariance is only guaranteed up to the resolution capability and scope of AST parsers, which the reviewer may have confused. In other words, if the ASTs of two queries are the same, then they are guaranteed to get the same representation in our pipeline. There's no guarantee otherwise, nor have we intended to claim it further.

For SQLs that involve different operators but may have equivalent semantic meanings under some conditions, there is no algorithmic "canonicalization" beyond the AST representation. The ASTs of an EXISTS query versus a JOIN + DISTINCT query will certainly be different. The"rewriting" is also not directly performed in the textual space, but implicitly in the neural embedding space. Instead, we leave that potential equivariance property for the graph neural network to automatically learn and resolve from the training data.

We would also like to point out that rewriting EXISTS as JOIN with DISTINCT is not always equivalent, and therefore we don't algorithmically map these two to the same representation. EXISTS tests whether each outer row has at least one match and preserves the multiplicity of qualifying outer rows. A JOIN can produce multiple rows for a single outer row. DISTINCT removes duplicates after projection, so it can remove those extra matches and also collapse separate outer rows that agree on the projected columns. Whether the two forms are equivalent depends on the join condition, the projected columns, and any relevant uniqueness guarantees. WARBLER therefore keeps the two operator structures distinct.

Question 2 — Controlled perturbations and user intent

We have followed the instructions to provide controlled evaluations. On BIRD dev, the pairwise analysis covers 3,366 pairs: syntactic edits, alternative SQL structures with matching outputs on the evaluated database, and single edits that change the output. The matching-output group measures agreement on the evaluated database instance.

The table reports mean absolute score changes (|ΔS|). The figure shows t-SNE projections of the paired embeddings - see Appendix D for the paired-score table and embedding diagram.

Rewrite / Perturbation CategoryPairsAverage Mean Score Change: Text Verifier (LLM w/o GNN)Average Mean Score Change: WARBLER (Ours)
1. Syntactic edits tested1,7840.0660.000
2. Output-matched alternative SQL structures1,1140.1400.188
3. Meaning-changing single edits4680.3510.388

In the reported comparison, WARBLER’s mean absolute score change is 0.000 on the sampled syntactic edits and 0.388 on meaning-changing edits. The output-matched alternatives show a mean change of 0.188.

We operationalize alignment with user intent through reference execution agreement. See §3.2 for the training labels and §4.1 for the execution accuracy metric.

Weakness 2 and Question 3 — Trained text selectors and related work

We would like to clarify a confusion in our writing: when we say “prompting the language model to score raw SQL text strings directly”, we actually mean the language model has been trained, just that the inputs do not contain graph (as reported below in “Trained SQL-Text Verifier”)

We have compiled a table below to report pairwise and pointwise text-selector comparisons on BIRD dev. WARBLER reaches 74.12% EX, 0.76 pp above the CHASE-SQL-style pairwise selector and 0.82 pp above the GradeSQL-style pointwise selector. See Appendix D, Table 7.

Candidate Selection Method (Same Pool & Labels)Scoring Complexity (# LLM Passes)BIRD Dev Accuracy
Trained raw-SQL ablationO(K)73.32%
Trained Pairwise Text Selector (CHASE-SQL style)O(K²)73.36%
Trained pointwise text selector (GradeSQL style)O(K)73.30%
WARBLER (Graph-Language Verifier)O(K)74.12%

Clarification on the specific contributions of WARBLER vs. CHASE-SQL / GradeSQL.

  • GradeSQL independently scores each candidate using a text-based outcome reward model. WARBLER also scores candidates independently and adds an embedding of the typed SQL graph.
  • CHASE-SQL compares candidates in pairs. WARBLER uses a pointwise score, so the number of candidate-scoring evaluations grows linearly with pool size.

The revised discussion of CHASE-SQL and GradeSQL appears in §5 under Candidate Selection and Code Verification.

“CHASE-SQL (Pourreza et al., 2025) generates diverse SQL candidates using three strategies: divide-and-conquer decomposition, reasoning from query execution plans, and instance-aware synthetic examples. At inference time, a fine-tuned pairwise selector compares candidates with different execution results, and the system selects the query with the most pairwise wins.

GradeSQL (Tritto et al., 2026) fine-tunes a pointwise outcome reward model on generated SQL candidates labeled by whether their execution results match those of the reference query. Given the natural-language question and database schema, the model estimates each candidate’s probability of correctness independently and selects the highest-scoring query.”

Thank you for the clarifications and additional experiments. The clarification of the AST-level guarantee and the trained text-selector comparisons address part of my concerns. However, a normalized-SQL-text baseline is still not clearly reported, leaving the benefit of graph encoding beyond normalization insufficiently established. Moreover, mean absolute score changes do not demonstrate whether scores move in the correct direction or whether final selection remains robust under the tested rewrites. These remaining gaps limit the evidence supporting the central contribution. I therefore keep my original rating.

We thank the reviewer for providing further feedback on our rebuttal experiments.

 

1. "Normalized-SQL-text"-only ablation

The reviewer may have misunderstood our "normalization" ("canonicalization") step - this has been explained by our reply to Weakness 1 "Scope of canonicalization":

  • Our normalization step cannot generate normalized SQL text, nor have we claimed it.
  • Instead, this step can only generates a SQL graph object consisting of typed nodes and edges, based on the AST-parsing function provided by sqlglot and some algorithmic node merging, as we have explained in Weakness 1 and Question 1.
  • For the SQL graph object to be consumed by our LLM-backboned verifier, the special graph object must first go through graph encoding.

Therefore, normalization and graph encoding are two steps that cannot be "disentangled" with one component ablated and the other kept as requested.

Meanwhile, our table already reports results with and without normalization. The "WARBLER (Graph-Language Verifier)" row reports the result with normalization. The "Trained raw-SQL ablation" row reports the result without normalization. The contrast between the two results show that the graph-based normalization pipeline improves performance.

2. Score movement direction & final selection

We agree that mean absolute score change measures sensitivity but does not preserve directional information. We have also evaluated individual candidates rather than the final choice from a candidate pool.

However, we wanted to highlight that our experiment has precisely addressed score consistency and stability under the tested syntactic rewrites, as requested in the original Question 2 -- whereas "directional change" is an additional aspect on top of the original request. Across the 1,784 pairs of synthetic rewrites, WARBLER’s mean absolute score change was 0.000 at the reported precision, compared with 0.066 for the trained text verifier -- in which smaller is better. Across the 468 edits that changed the execution result, the corresponding changes were 0.388, versus 0.351 by the baseline -- in which larger is better. This shows that WARBLER’s scores were stable under the tested syntactic rewrites and sensitive to edits that changed execution results.

Reviewer_fhjpOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose WARBLER, a two-stage framework combining an execution-guided candidate generator and a learned Graph-Language Verifier. The verifier parses SQL queries into typed DAGs, encodes them via a GIN, and fuses the graph representation with LLM context embeddings to predict query correctness. On the BIRD benchmark, WARBLER achieves 80.04% execution accuracy on the official test set.

Critical strengths
  1. Reaching 80.04% Execution Accuracy on the official BIRD held-out test set is an impressive benchmark milestone for a single-model pipeline, outperforming prior top systems

  2. The paper clearly articulates why sequential LLM scoring fails for SQL verification—namely, sensitivity to surface formatting and failure to recognize structural equivalence.

Critical weaknesses
  1. The core technical concept of compiling generated SQL candidates into standardized graph or AST units to eliminate surface syntactic noise, and training a graph-augmented verifier/ranker to align schema/question intent with query logic, is conceptually identical to SGU-SQL, i.e., structure-guided SQL.

  2. Since the verifier learns to score candidates generated by a specific LLM backbone, it remains unclear whether the verifier learns general relational logic alignment or merely overfits to the idiosyncratic mistake patterns of Gemini 3.1 Pro.

  3. If candidate queries are generated by an open-source model, e.g., Qwen-Coder or Llama-3 or a different frontier model, does the WARBLER verifier retain its accuracy advantage?

  4. The candidate generation samples K=7 queries per question, followed by AST parsing, GNN forward pass, LLM context encoding, and candidate scoring. The paper lacks a breakdown of the latency and computational overhead, e.g., wall-clock time per query compared to standard single-pass generation or execution self-consistency.

Questions for authors

Please see above.

1 comment

Thank you for your feedback on WARBLER’s novelty, transfer across generators, and runtime.

Weakness 1 — Comparison with SGU-SQL

We respectfully clarify that WARBLER and SGU-SQL are fundamentally different:

  • SGU-SQL uses graphs to model the natural-language question and database schema, then uses the question’s syntax tree to guide SQL generation – in the pre-generation stage.It also does NOT have a graph-language neural model that can directly consume the graph.
  • WARBLER uses graphs to model generated SQL candidates – in the post-generation stage. It further encodes this graph with a graph-language model and combines it with the question, schema, and evidence to score the candidate. See §5 (Graph Representations and Graph-Language Alignment).

Weakness 2 — Generalization beyond Gemini 3.1 Pro

The verifier learns both.

We add an analysis to check where the improvement comes from: please check our Figure 6 in Appendix B for the categorical analysis of where the improvement comes from.

Among 396 incorrect SQL candidates in our, our verifier successfully identified and avoided 111 general relational-logic errors, and 31 instances of recurring idiosyncratic mistakes, accounting for 78.2% and 21.8% of the 142 classified errors avoided, respectively.

Meanwhile, we also would like to highlight the importance of avoiding idiosyncratic mistakes, and our verifier does not overfit to a particular LLM backend we use. Evidence is shown in Question 3’s answer where the generator and verifier use different LLMs: A less capable LLM for generator will create more idiosyncratic mistakes, but since our verifier is still able to recover a significant portion of it and boost the score up to a relatively competitive edge.

Weakness 3 — Transfer to other generators

Yes. We keep the Gemini 3.1 Pro–trained verifier fixed and change the generator. The results are reported below. WARBLER improves over expected Pass@1 by 1.19 pp for Llama 4 Maverick and Gemma 4 31B, 1.77 pp for GPT-OSS-120B, and 1.51 pp for Qwen3-Coder-30B-A3B-Instruct. See Appendix C, Table 3.

Generator BackboneSingle pass (K=1)Expected Pass@1WARBLER (K=7)Gain vs. expected Pass@1Oracle Pass@7
Llama-4-Maverick-17B-128E61.80%62.02%63.21%+1.19 pp65.58%
GPT-OSS-120B58.41%58.11%59.88%+1.77 pp65.84%
Gemma-4-31B-IT66.88%67.10%68.29%+1.19 pp70.08%
Qwen3-Coder-30B-A3B-Instruct55.80%56.73%58.24%+1.51 pp63.37%
Gemini 3.1 Pro (ours)72.33%72.45%74.12%+1.67 pp77.01%

Expected Pass@1 is the expected execution accuracy from choosing one candidate uniformly at random from each question’s pool. We compute it by averaging the fraction of execution-correct candidates in each pool across questions.

Oracle Pass@7 is the percentage of questions with at least one execution-correct candidate in the pool generated with K=7K=7. It gives the maximum accuracy a selector could achieve on those pools.

Weakness 4 — Wall-clock timing

Please find the wall-clock time report below, per question on BIRD Dev, K=7. We note that our stage-2 verification overhead is negligible, because this stage only involves prefill rather than autoregression. See Appendix C, Tables 4 and 5.

Pipeline stageWall-Clock TimeShare of Total
Stage 1a: Candidate Generation3.361 s87.03%
Stage 1b: SQL execution0.423 s10.95%
Stage 2a: SQL-to-Graph Parsing0.032 s0.83%
Stage 2b: LLM context encoding0.021 s0.54%
Stage 2c: candidate scoring0.025 s0.65%
Stage 2 Subtotal (Total Verifier Overhead)0.078 s2.02%
SystemWall-clock timeBIRD dev EX (%)
WARBLER (sum of stages)3.862 s74.12%
Baseline: Standard Single-Pass Generation2.351 s72.40%
Baseline: Execution Self-Consistency3.938 s73.02%

Comments on the paper

1 comment

Dear Area Chairs and Reviewers,

Thank you for your careful reviews. We appreciate your recognition of WARBLER’s 80.04% execution accuracy on the official BIRD test set, 2.90 percentage points above the previous best on the single-model track, and of the motivation for representing generated SQL as graphs. The responses below summarize what we clarified and what the additional experiments show.

  • Novelty (fhjp W1, sczU W2/Q3). SGU-SQL builds question and schema graphs to guide generation. WARBLER builds graphs of generated SQL to score candidates. We clarify that the raw-SQL ablation uses supervised training and compare WARBLER with CHASE-SQL-style pairwise and GradeSQL-style pointwise selectors.
  • Transfer across generators (fhjp W2/W3). Our BIRD dev error analysis covers general relational errors and recurring database-specific mistakes. When fixing our verifier, WARBLER improves over expected Pass@1 by 1.19% – 1.77% on pools from four open-weight generators.
  • “Canonicalization” (sczU W1/Q1/Q2). We distinguish invariance of the graph embedding from invariance of the final score and explain why EXISTS and JOIN with DISTINCT can differ. The 3,366-pair analysis measures score changes.
  • Latency (fhjp W4). On BIRD dev, verification takes 78 ms per question, or 2.02% of the stage total. We report the stage breakdown and the single-pass and execution self-consistency baselines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.