- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents WARBLER, a Text-to-SQL framework combining execution-guided candidate generation and repair with a learned graph-language verifier. Candidate SQL queries are encoded into typed directed acyclic graphs, encoded using a graph neural network, and combined with contextual language representations to predict execution correctness. The highest-scoring candidate is selected as the final prediction. The paper reports 80.04% execution accuracy on the BIRD test set and development-set improvements over execution self-consistency and text-only LLM scoring.
Critical strengths
- The paper reports 80.04% execution accuracy on the BIRD held-out test set, outperforming the strongest listed baseline by 2.90 percentage points.
- Representing SQL as typed relational graphs makes operators, entities, and their dependencies explicit. This provides a well-motivated structural inductive bias for interpreting SQL and assessing its compatibility with the natural-language question.
Critical weaknesses
-
A central argument is that semantically equivalent SQL queries are compiled into isomorphic graphs, leading to invariant representations and scores. However, Section 3.2 mainly describes column-to-table resolution, alias standardization, and node deduplication. It does not clearly define the supported rewrite classes or explain how structurally different but semantically equivalent implementations are canonicalized. For example, an EXISTS query and a JOIN followed by DISTINCT can express the same query under suitable constraints while containing different operators and dependencies.GNN permutation invariance does not establish invariance across these alternative SQL implementations; the latter depends on the compiler producing compatible representations. Conversely, queries that happen to return the same result on one database need not be semantically equivalent and should not necessarily be collapsed.
-
WARBLER is a pointwise learned verifier used to select among generated candidates. Closely related methods already investigate learned SQL selection. CHASE-SQL (ICLR 2025) combines candidate generation and repair with a trained pairwise selector, while GradeSQL trains outcome reward models using execution-based correctness labels to score and rank SQL candidates. These approaches are not discussed or directly compared in the submission.The graph representation could still provide a meaningful contribution, but the current ablations do not isolate that contribution. The text-only baseline in Section 4.3 is described as prompting an LLM to score SQL, whereas WARBLER receives supervised training. Thus, this comparison changes both the representation and the training procedure. Removing the parser and GNN together also leaves the effects of normalization and graph encoding entangled.
Questions for authors
-
What SQL rewrite classes and database assumptions are covered by the claimed canonicalization guarantee? How does the compiler handle structurally different but equivalent implementations, such as EXISTS versus JOIN with DISTINCT? Please clarify whether the guarantee is restricted to a defined subset of rewrites.
-
Provide controlled evaluations of equivalent rewrites and meaning-changing perturbations, measuring both score consistency and stability of the final candidate selection. Is “alignment with user intent” operationally defined entirely through reference execution agreement, or is there additional evidence supporting a broader interpretation?
-
Compare the proposed verifier with trained text-based selectors using the same candidate pool and correctness labels, including a normalized-SQL-text baseline. Please also clarify the specific contribution relative to existing learned selection methods such as CHASE-SQL and GradeSQL.
4 comments
Weakness 2 and Question 3 — Trained text selectors and related work
We would like to clarify a confusion in our writing: when we say “prompting the language model to score raw SQL text strings directly”, we actually mean the language model has been trained, just that the inputs do not contain graph (as reported below in “Trained SQL-Text Verifier”)
We have compiled a table below to report pairwise and pointwise text-selector comparisons on BIRD dev. WARBLER reaches 74.12% EX, 0.76 pp above the CHASE-SQL-style pairwise selector and 0.82 pp above the GradeSQL-style pointwise selector. See Appendix D, Table 7.
| Candidate Selection Method (Same Pool & Labels) | Scoring Complexity (# LLM Passes) | BIRD Dev Accuracy |
|---|---|---|
| Trained raw-SQL ablation | O(K) | 73.32% |
| Trained Pairwise Text Selector (CHASE-SQL style) | O(K²) | 73.36% |
| Trained pointwise text selector (GradeSQL style) | O(K) | 73.30% |
| WARBLER (Graph-Language Verifier) | O(K) | 74.12% |
Clarification on the specific contributions of WARBLER vs. CHASE-SQL / GradeSQL.
- GradeSQL independently scores each candidate using a text-based outcome reward model. WARBLER also scores candidates independently and adds an embedding of the typed SQL graph.
- CHASE-SQL compares candidates in pairs. WARBLER uses a pointwise score, so the number of candidate-scoring evaluations grows linearly with pool size.
The revised discussion of CHASE-SQL and GradeSQL appears in §5 under Candidate Selection and Code Verification.
“CHASE-SQL (Pourreza et al., 2025) generates diverse SQL candidates using three strategies: divide-and-conquer decomposition, reasoning from query execution plans, and instance-aware synthetic examples. At inference time, a fine-tuned pairwise selector compares candidates with different execution results, and the system selects the query with the most pairwise wins.
GradeSQL (Tritto et al., 2026) fine-tunes a pointwise outcome reward model on generated SQL candidates labeled by whether their execution results match those of the reference query. Given the natural-language question and database schema, the model estimates each candidate’s probability of correctness independently and selects the highest-scoring query.”
Thank you for the clarifications and additional experiments. The clarification of the AST-level guarantee and the trained text-selector comparisons address part of my concerns. However, a normalized-SQL-text baseline is still not clearly reported, leaving the benefit of graph encoding beyond normalization insufficiently established. Moreover, mean absolute score changes do not demonstrate whether scores move in the correct direction or whether final selection remains robust under the tested rewrites. These remaining gaps limit the evidence supporting the central contribution. I therefore keep my original rating.
We thank the reviewer for providing further feedback on our rebuttal experiments.
1. "Normalized-SQL-text"-only ablation
The reviewer may have misunderstood our "normalization" ("canonicalization") step - this has been explained by our reply to Weakness 1 "Scope of canonicalization":
- Our normalization step cannot generate normalized SQL text, nor have we claimed it.
- Instead, this step can only generates a SQL graph object consisting of typed nodes and edges, based on the AST-parsing function provided by sqlglot and some algorithmic node merging, as we have explained in Weakness 1 and Question 1.
- For the SQL graph object to be consumed by our LLM-backboned verifier, the special graph object must first go through graph encoding.
Therefore, normalization and graph encoding are two steps that cannot be "disentangled" with one component ablated and the other kept as requested.
Meanwhile, our table already reports results with and without normalization. The "WARBLER (Graph-Language Verifier)" row reports the result with normalization. The "Trained raw-SQL ablation" row reports the result without normalization. The contrast between the two results show that the graph-based normalization pipeline improves performance.
2. Score movement direction & final selection
We agree that mean absolute score change measures sensitivity but does not preserve directional information. We have also evaluated individual candidates rather than the final choice from a candidate pool.
However, we wanted to highlight that our experiment has precisely addressed score consistency and stability under the tested syntactic rewrites, as requested in the original Question 2 -- whereas "directional change" is an additional aspect on top of the original request. Across the 1,784 pairs of synthetic rewrites, WARBLER’s mean absolute score change was 0.000 at the reported precision, compared with 0.066 for the trained text verifier -- in which smaller is better. Across the 468 edits that changed the execution result, the corresponding changes were 0.388, versus 0.351 by the baseline -- in which larger is better. This shows that WARBLER’s scores were stable under the tested syntactic rewrites and sensitive to edits that changed execution results.
Thank you for your careful review and for asking us to clarify the scope of canonicalization and the verifier comparisons.
Weakness 1 and Question 1 — Scope of canonicalization
We want to clarify that the nice structural equivariance is only guaranteed up to the resolution capability and scope of AST parsers, which the reviewer may have confused. In other words, if the ASTs of two queries are the same, then they are guaranteed to get the same representation in our pipeline. There's no guarantee otherwise, nor have we intended to claim it further.
For SQLs that involve different operators but may have equivalent semantic meanings under some conditions, there is no algorithmic "canonicalization" beyond the AST representation. The ASTs of an
EXISTSquery versus aJOIN+DISTINCTquery will certainly be different. The"rewriting" is also not directly performed in the textual space, but implicitly in the neural embedding space. Instead, we leave that potential equivariance property for the graph neural network to automatically learn and resolve from the training data.We would also like to point out that rewriting
EXISTSasJOINwithDISTINCTis not always equivalent, and therefore we don't algorithmically map these two to the same representation.EXISTStests whether each outer row has at least one match and preserves the multiplicity of qualifying outer rows. AJOINcan produce multiple rows for a single outer row.DISTINCTremoves duplicates after projection, so it can remove those extra matches and also collapse separate outer rows that agree on the projected columns. Whether the two forms are equivalent depends on the join condition, the projected columns, and any relevant uniqueness guarantees. WARBLER therefore keeps the two operator structures distinct.Question 2 — Controlled perturbations and user intent
We have followed the instructions to provide controlled evaluations. On BIRD dev, the pairwise analysis covers 3,366 pairs: syntactic edits, alternative SQL structures with matching outputs on the evaluated database, and single edits that change the output. The matching-output group measures agreement on the evaluated database instance.
The table reports mean absolute score changes (|ΔS|). The figure shows t-SNE projections of the paired embeddings - see Appendix D for the paired-score table and embedding diagram.
In the reported comparison, WARBLER’s mean absolute score change is 0.000 on the sampled syntactic edits and 0.388 on meaning-changing edits. The output-matched alternatives show a mean change of 0.188.
We operationalize alignment with user intent through reference execution agreement. See §3.2 for the training labels and §4.1 for the execution accuracy metric.