acceptodds
Under review as a conference paper at ICLR 2027

Derivation-SQL: A Cost-Effective Graph-Based Framework for Text-to-SQL

Abstract

Text-to-SQL enables users to query relational databases in natural language and has broad applications. However, existing retrieval approaches based on question semantics may ignore the derivation relationships among SQL queries. In contrast, structure-based approaches that use predicted SQL require an additional generation step, increasing inference cost. To address these issues, we propose Derivation-SQL, a cost-effective Text-to-SQL framework that uses SQL derivation graphs to retrieve structurally relevant examples directly from natural language questions. We first construct an SQL derivation graph (DG) by decomposing training SQL queries into subqueries and connecting queries through derivation relationships. This process also expands the pool of few-shot candidates. We then design a DG-based embedding network (DGE) that encodes text semantics and SQL derivation relationships. The network encodes examples offline and maps natural language questions to the same embedding space online. A tailored pipeline with improved prompts seamlessly integrates these components into an end-to-end solution. Experiments on BIRD and Spider show that Derivation-SQL achieves competitive execution accuracy with low token consumption, reaching a Pareto-optimal balance between accuracy and cost. With GPT-4o, Derivation-SQL achieves execution accuracy of 72.56% on BIRD-Dev, which costs less than half the token usage of other representative approaches.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text-to-SQL, NL2SQL, Large Language Model, Natural Language Processing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rGqAOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Derivation-SQL, a graph-based framework that leverages derivation relationships among SQL queries for few-shot example retrieval in Text-to-SQL. The method decomposes training SQL queries to expand the example pool and constructs a SQL Derivation Graph (DG) that connects examples through these relationships. A Derivation Graph Embedding Network (DGE) then learns representations that incorporate question semantics and SQL derivation relationships, enabling retrieval directly from natural-language questions without preliminary SQL generation. Experiments on BIRD and Spider report competitive execution accuracy, with reduced token usage and inference costs in the reported comparisons.

Critical strengths

Strength 1. Strong performance with efficient inference.

The method achieves competitive execution accuracy on BIRD and Spider, with a promising balance between accuracy and inference cost. Offline graph construction and example encoding enable retrieval without preliminary SQL generation. With GPT-4o, it reports 72.56% execution accuracy on BIRD-Dev at an average cost of $0.031 per query.

Strength 2. An interesting use of SQL decomposition for example retrieval.

The idea of decomposing training SQL queries to construct additional examples and model their derivation relationships is interesting. Rather than treating these examples independently, the method uses their relationships to guide representation learning, connecting example augmentation with graph-based retrieval in a coherent framework.

Critical weaknesses

Weakness 1. Limited Novelty and Insufficient Differentiation from Prior Work

  • The problem formulation is not clearly distinguished from prior work. The limitations of semantic retrieval and the cost of preliminary SQL generation have already been explored. DCG-SQL [1] addresses similar challenges, while ReFSQL [2] also proposes structure-aware question representations. The differences from prior work should be more clearly articulated.
  • The paper does not sufficiently explain which limitations of existing methods remain unresolved or why modeling SQL derivation relationships is necessary. A clearer discussion of ReFSQL and DCG-SQL would better establish its unique contributions.
  • Prior approaches such as GAT [3] and RAT-SQL [4] have already explored incorporating graph structures or relational information into attention mechanisms. The paper should clarify whether its encoder introduces novel architectural components or, if it builds on existing techniques, better explain the motivation for adopting them in the context of SQL derivation graphs.
  • The paper lacks direct comparisons with existing structure-aware retrieval methods that avoid preliminary SQL generation. Such comparisons are necessary to demonstrate the specific effectiveness and efficiency advantages of the proposed approach.

Weakness 2. Limited analysis of the methodology

  • For DGE, the motivation for the hybrid encoder is insufficiently explained. Although lines 275–279 describe the encoder, the discussion focuses primarily on implementation details rather than explaining why a separate Hybrid encoder is necessary.
  • In addition, the paper lacks a dedicated ablation study that isolates the contribution of DG. Although DG is presented as one of the paper’s core contributions, it remains unclear how much it contributes to the reported performance improvements. Further analysis is needed to determine how the extent of few-shot example augmentation through SQL decomposition affects downstream performance.
  • Several design choices lack sufficient explanation, particularly the concatenation of mean-pooled representations from the last three BERT layers. Further discussion or ablation studies would help clarify why these specific choices were made and what benefits they provide.

[1] Lee, J., et al. DCG-SQL: Enhancing In-Context Learning for Text-to-SQL with Deep Contextual Schema Link Graph

[2] Zhang, K., et al. ReFSQL: A Retrieval-Augmentation Framework for Text-to-SQL Generation

[3] Veličković, P., et al. Graph Attention Networks. International Conference on Learning Representations

[4] Wang, B., et al. RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers

Questions for authors

Question 1. Quality and diversity of augmented few-shot examples.

Although Appendix E.6 evaluates the quality of the generated questions, I would like to understand what additional structural diversity the augmented examples introduce and how much they contribute to downstream performance. Could the authors provide a quantitative analysis of this diversity, such as coverage of SQL patterns or operation combinations, and assess the performance gains attributable to example augmentation? A controlled comparison between the original, non-augmented candidate pool and the augmented pool, using the same retrieval method and downstream pipeline, would help clarify the effectiveness of the augmentation.

Question 2. Contribution of graph structure.

The proposed DG offers an interesting way to model SQL derivation relationships. However, I would like to understand how much additional benefit the graph structure provides beyond expanding the few-shot candidate pool. Could the authors isolate the contribution of graph connectivity by comparing the proposed method with a variant that uses the same augmented examples but without graph-based encoding, while keeping the remaining experimental settings fixed?

4 comments

Thank you for the constructive feedback!

Considering cost and efficiency, we conduct the experiment using the Deepseek-v3 as the LLM backbone.

For W1:

1 & 2: We clarify that our work is different from DCG-SQL and ReFSQL. As discussed in the introduction and related work, Derivation-SQL leverages SQL decomposition to construct a larger pool of fine-grained few-shot examples for retrieval, which DCG-SQL and ReFSQL do not possess. These decomposed examples offer more granular guidance for LLMs. To retrieve few-shot examples more effectively, Derivation-SQL designs a novel retrieval method tailored to the characteristics of derivation examples, which enables retrieval beyond vanilla semantic matching. Derivation-SQL features a fundamentally different graph construction: it models derivation relationships among SQL queries, in contrast to DCG-SQL and ReFSQL, which prioritize schema relationships and token-level semantics between questions and database schemas.

Technically, there are several key differences. First, the graph construction is different. DCG-SQL and ReFSQL construct graphs based on question–schema and token semantic relations, while our derivation graph captures SQL-to-SQL derivation dependencies. Second, the encoding pipeline is different. Derivation-SQL is more efficient than baselines. DCG-SQL and ReFSQL need to use graph neural networks. In contrast, Derivation-SQL encodes nodes offline. During online inference, Derivation-SQL encodes the question without graph computation and uses dot-product similarity for node retrieval. Third, DCG-SQL and ReFSQL require few-shot examples and the target schema to belong to the same database, since their graph nodes for questions and schemas must interact. This constraint limits their applicability on benchmarks like BIRD, where training and dev databases are disjoint. Our method removes this restriction.

Notably, although DCG-SQL and ReFSQL introduce structure-aware representations, they do not model how SQL queries are derived from one another. This derivation signal is the core unresolved limitation we target: semantic or question-schema matching does not explicitly model the derivation paths between SQL variants, which are valuable for guiding complex SQL generation.

3: GAT is a general graph attention architecture and is not tailored for our derivation graph. RAT-SQL is a structure-aware method designed for question–schema modeling, which focuses on capturing schema connections and alignments between questions and database columns. In contrast, our work targets derivation relationships among SQL queries. Therefore, neither method can be directly adopted for our derivation graph.

4: We compare our DGE with the structure-enhanced retriever (SER) from ReFSQL. Following the setup of ReFSQL, we train BERT via contrastive learning to encode both questions and graph nodes. SQL queries with high tree edit distances are treated as positive samples, and others as negatives.

LLMDGESER
GPT-4o73.0963.12
Gemini-2.5-flash71.1068.44

As shown in the table, the performance of DGE is better than that of SER under both GPT-4o and Gemini-2.5-flash.

For W2:

1: The motivation for the hybrid encoder is to integrate text-based information and SQL-based information, so as to provide a new aspect to retrieve examples. The hybrid encoder could help retrieve different examples, making the retrieval more robust.

2: To verify the effectiveness of the DG, we conduct experiments by using DG or not. When removing DG, we use the training set as the few-shot example provider and use Bert to encode the question of the training set.

MethodsEX
Derivation-SQL66.62
DGE -> Bert64.21
DGE -> Bert*64.8
w/o DG64.67
w/o DG*65.25

From the table, we find that: First, directly using DG without DGE, the performance is weaker than that of using the original training set. This is because Bert cannot capture the structure information, while the similar expression of questions misleads BERT.

3: The concatenation of mean-pooled representations from the last three BERT layers is a common trick for improving performance. Because each layer of Bert captures different information, mean-pooled representations from the last three BERT could provide more information for subsequent module.

For Q1:

We supplement a diversity analysis of DG which including both the training set and the derived data. Values inside parentheses denote the proportion of samples within the corresponding data. We count the number of samples with 1, 2, 3, or (>= 4) JOIN clauses, as well as samples with 1, 2, 3, 4, or (>= 5) query predicates. Nested Queries and Aggregate Functions report the count of samples that include at least one nested query or aggregate function, respectively. Compared with the distribution of the training set, the number of each type SQL queries in the DG have significant improvement.

DataSizeJOIN (1/2/3/>=4)Query Predicates (1/2/3/4/>=5)Nested QueriesAggregates Function
Training set94283038 (32.22) /4683 (49.67) /1355 (14.37)/352 (3.73)1212 (12.86)/1520 (16.12)/2839 (30.11)/2097 (22.24)/1760 (18.67)633 (6.71)4002 (42.45)
DG250698207 (32.74)/12046 (48.05)/3749 (14.95)/1067 (4.26)4072 (16.24)/4954 (19.76)/7465 (29.78)/4898 (19.54)/3680 (14.68)1138 (4.54)5416 (21.6)

To quantify the performance gain brought by this augmentation, we refer to the controlled ablation in W2: under the same retrieval and LLM pipeline, the model with the augmented DG pool achieves 66.62% EX, while the non-augmented setting (w/o DG, using only the original training set as candidates) reaches 64.67% EX. This demonstrates that the structurally diverse augmented examples bring consistent performance improvements.

For Q2:

To isolate the contribution of graph connectivity, we conduct additional ablation experiments on BIRD-Dev. All variants share the same augmented candidate examples from DG; we only modify the derivation edges while keeping all other settings fixed.

MethodsEX
Derivation-SQL66.62
No edges63.95
Random edges64.93

The results show that the graph structure with authentic derivation edges brings clear performance gains. Removing all derivation edges leads to a substantial EX drop. Randomly shuffled edges yield better performance than the edge-free variant but still underperform the original DG, which demonstrates that the true SQL derivation connectivity, rather than arbitrary graph connections, is essential for our method.

Thank you for your detailed response. I encourage the authors to incorporate the clarifications and additional experimental results into the revised paper. I will raise my score.

Thank you for your careful review and for acknowledging our contributions. We will incorporate all these clarifications and additional experimental results into the revised paper.

Reviewer_WtxhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Derivation-SQL, which builds a derivation graph by decomposing SQL queries in the training set.

It trains a graph embedding network (DGE) to map natural-language questions directly into a representation space that encodes SQL structural information for few-shot example retrieval. The method does not require generating SQL before retrieval.

Experiments are conducted on BIRD and Spider. With GPT-4o, the method achieves 72.56% execution accuracy on BIRD-Dev at an average cost of $0.031 per question.

Critical strengths
  1. The method learns SQL structural information offline while retaining vector-based example retrieval at inference time. This avoids the additional generation overhead incurred by structure-based retrieval methods that first predict SQL.

  2. Appendix Table 7 compares DGE with semantic, SQL-structure, and AST-based retrieval methods under the same downstream pipeline. Fig. 7 further shows improvements when DGE is integrated into existing Text-to-SQL systems. These results support the practical effectiveness of DGE.

Critical weaknesses
  1. Although Fig. 6 and Appendix Table 7 provide relevant ablations and retrieval comparisons, they do not fully isolate the contribution of the derivation graph structure. The method introduces SQL decomposition, additional question–SQL pairs, and graph-based representation learning simultaneously. A controlled comparison that removes or randomizes graph edges while keeping the candidate pool and encoder unchanged would help determine whether the gains specifically come from modeling SQL derivation relations.

  2. The cost comparisons are not fair, and the main efficiency claim is insufficiently supported. The baselines in Tables 1 and 3 use different backbone models and experimental settings, making it difficult to attribute cost differences to the method itself. Some reported results are also inconsistent: CHASE-SQL achieves 73.01% and 74.46% in the two tables, respectively, while Alpha-SQL achieves 69.7% and 71.2%. The paper does not explain these discrepancies. Moreover, the cost analysis mainly considers online inference and does not report amortized costs for offline question generation and DGE training. Therefore, the current experiments are insufficient to support the claim of Pareto optimality in cost and accuracy.

Questions for authors
  1. Under the same candidate example pool, generation model, and prompts, what is the performance when decomposed SQL queries are included without using graph structure? How does performance change if graph edges are removed or randomly rewired?

  2. Derivation edges are directed, but link prediction uses symmetric dot-product similarity. How does the model learn the direction of derivation? In addition, Appendix C.3 states that approximately 3.25% of the decomposed SQL queries are not executable. Does removing these nodes affect performance?

  3. The inconsistent baseline accuracies in Tables 1 and 3 need further explaining. Could the authors compare methods under the same backbone model and cost-accounting conventions, and report average cost per query after amortizing the offline costs?

3 comments

Thank you for the constructive feedback!

W1 & Q1

All four settings were evaluated on the full BIRD-Dev set, with the augmented candidate pool, generation model (DeepSeek-V3), and prompts held fixed. DGE->BERT encodes the derived SQL examples with BERT without graph structure. In the random-edge variant, each seed node connects to 10 randomly sampled nodes, matching the value of the maximum subgraph-size parameter.

MethodsEX
Derivation-SQL66.62
No edges63.95
Random edges64.93
DGE -> BERT64.21

Derivation-SQL achieves the highest EX of 66.62%, outperforming DGE->BERT, the no-edge method, and the random-edge method by 2.41, 2.67, and 1.69 percentage points, respectively. With the same augmented candidate pool, generation model, and prompts, these results support the benefit of graph-based encoding, while the edge ablations further highlight the value of the original graph structure beyond candidate-pool augmentation.

Q2

The direction of edges is learned via the directed adjacency matrix. In the matrix, the aija_{ij} = 1 if and only if there exists a directed edge from aia_i to aja_j. We acknowledge that vanilla dot-product similarity is symmetric by itself. However, our model is trained with directed information from the adjacency matrix, which drives the node embeddings to capture directional patterns for ordered node pairs (aia_i, aja_j). Consequently, the dot product between embeddings of aia_i and aja_j estimates the likelihood of a directed edge aia_i -> aja_j, rather than aja_j -> aia_i.

For the example retrieval, we retain instead of discarding incorrect SQL queries. We conduct an ablation experiment on DGE training to examine the performance impact of removing incorrect SQL nodes from the DG.

MethodsEX
Derivation-SQL66.62
Removing incorrect nodes64.02

As shown in the table, removing incorrect nodes causes a 2.6% performance decline. This demonstrates that incorrect SQL nodes still provide useful information for DGE training.

W2 & Q3

We apologize for the inconsistent reporting of CHASE-SQL and Alpha-SQL. CHASE-SQL’s 73.01% comes from its original paper, whereas 74.46% comes from the leaderboard; we will use the paper result consistently. Alpha-SQL’s 69.7%/71.2% use Qwen2.5-Coder-32B/Qwen3-Coder-30B-A3B, respectively. We will label these configurations separately and pair each accuracy with its corresponding cost.

For practical deployment, we compare full GPT-4o pipelines under their reported settings. For consistent accounting, we normalize online costs at USD 2.50/M input tokens and USD 10/M output tokens, rounded to three decimal places:

MethodEXInput tokens (K)output tokens (K)total tokens (K)Reported online cost ($/query)
RSL-SQL67.2114.220.4714.690.04
Opensearch-SQL69.3022.034.7326.760.102
E-SQL65.6441450.12
Derivation-SQL72.5611.050.3311.380.031

Within this comparison, Derivation-SQL achieves the highest EX and lowest online cost: +3.26 EX points and approximately 70% lower cost than OpenSearch-SQL.

We note that our claim of Pareto optimality is explicitly defined with respect to online inference cost. Regarding the offline cost you raised, we provide the following clarifications. First, question generation incurs modest overhead. Offline question generation runs once and is reused during inference. Using the reported per-question averages, generating 15,641 questions with Gemini-2.5-flash requires approximately 29.4 million input tokens and 1.4 million output tokens. As the volume of online Text-to-SQL requests grows, the amortized offline cost per call becomes negligible, since online requests in practice far exceed our 1,500 test queries. Second, DGE training also incurs low overhead. With only 60M parameters, DGE has limited training cost. The table reports online inference cost. Over N queries served without rebuilding, the average total cost is 0.031 + C_offline/N, where C_offline includes question generation, DGE training, and graph/index construction.

[1] OpenSQL: Data-Efficient Text-to-SQL for Open-Source LLMs via Synthesized Intermediate Supervision, VLDB 2026

Thank you for the detailed response and additional experiments.

The newly added ablation experiments have effectively addressed my concerns about the contribution of the graph structure, and the explanations for the baseline result differences and the online inference cost have also improved.

However, there are still two issues that have not been resolved. One is that I do not think the offline cost can be ignored because it is a framework that depends on the deployment scale. The other is that the modeling of directed edges still has problems. I believe this only indicates that the node representations may contain directionally-related context, but it cannot solve the symmetry problem of the final prediction scores. Unless the authors actually adopt different source node and target node representations or asymmetric decoders, the current explanations are mathematically insufficient. I therefore maintain my original rating.

Thank you for your follow-up.

Offline cost: including it, Derivation-SQL remains the most cost-effective among the compared GPT-4o pipelines. We agree that the offline cost should be included. It is a one-time cost that depends on the training set size rather than the deployment scale, and the DG does not need to be rebuilt for unseen databases (BIRD-Dev databases are disjoint from the training databases). Question generation with Gemini-2.5-flash costs about 12.3 in total (0.30/M × 29.4M input tokens + 2.50/M × 1.4M output tokens). Amortized over the 1,534 BIRD-Dev questions, this adds 0.008 per query, giving a total cost of 0.039 per query with GPT-4o. This is well below OpenSearch-SQL (0.102) and E-SQL (0.12), and comparable to RSL-SQL (0.040), over which Derivation-SQL achieves 5.35 pp higher EX. We will report the amortized cost in the revision.

Directed edges: the symmetry concerns an auxiliary training loss rather than retrieval. We agree that the dot-product link-prediction score is symmetric and cannot by itself distinguish edge direction although we have utilized the direction information in node encoding. Direction information enters only through the directed adjacency mask in the node encoders, and the main retrieval task scores question–example relatedness, which does not require directional scores. Dot-product scoring also allows the offline-encoded nodes to be indexed with approximate nearest neighbor search (ANNS), which keeps online retrieval fast. Our empirical claim is unaffected: under identical settings, authentic derivation edges outperform no edges and random edges (66.62% vs. 63.95% and 64.93% EX). We will describe link prediction as modeling derivation relatedness in the revision and explore asymmetric decoders as an extension.

We will incorporate both points to improve our cost reporting and method description, while the main results, namely higher accuracy at lower cost and the contribution of derivation edges, remain unchanged.

Reviewer_QKJkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Derivation-SQL to improve the structural relevance of few-shot example retrieval for Text-to-SQL while reducing online generation costs. Its core idea is to construct a derivation graph by recursively decomposing SQL queries in the training set, connecting nodes through derivation relationships from simpler to more complex queries, and generating natural-language questions for newly introduced subqueries. The authors then train a Derivation Graph Embedding network (DGE) using three representations—text, SQL, and their fusion—to map user questions directly into the example retrieval space, avoiding an additional SQL generation step before online retrieval. The complete system also includes schema linking, example reranking, three-path SQL generation, error correction, and candidate selection.

The authors evaluate the method on BIRD and Spider. With GPT-4o, the paper reports an execution accuracy of 72.56% on BIRD-Dev, with an average consumption of 11.38K tokens per question. With Gemini-2.5-pro, it achieves an execution accuracy of 88.93% on Spider-Test. The paper also presents component ablations, retriever comparisons under a fixed generation pipeline, and experiments integrating DGE into other systems. Overall, the results demonstrate a valuable trade-off between accuracy and online cost, although stronger independent validation of the graph structure’s contribution and greater consistency in cost accounting are still needed.

Critical strengths
  1. Moving the main computational burden of structure-aware retrieval offline has clear practical value. Through the derivation graph and joint text–SQL representations, the method incorporates SQL structural information into retrieval representations of natural-language questions, allowing online example retrieval without first generating SQL. This design addresses a practical tension between structural relevance and online computational cost. Its value lies in improving the cost trade-off between retrieval and generation, without requiring state-of-the-art execution accuracy.

  2. The experiments include meaningful controlled comparisons of retrievers. Appendix Table 7 compares random, semantic, SQL-structure, AST, and DGE retrieval under a fixed generation pipeline. With GPT-4o, DGE achieves an execution accuracy of 73.09%, exceeding the best result among the other retrievers in the table, 68.11%. With Gemini-2.5-flash, it also improves accuracy from 70.10% with semantic retrieval to 71.10%. These results provide stronger evidence for the retrieval module’s effectiveness than comparisons aggregated across papers. Experiments integrating DGE into OpenSearch-SQL and CHESS also provide preliminary evidence of the module’s transferability.

Critical weaknesses
  1. The effects of the SQL derivation graph, the paper’s core contribution, have not been sufficiently disentangled from those of data augmentation and retriever training. The full method simultaneously introduces SQL subquery augmentation, generated question annotations, a trainable retriever, graph-neighbor interactions, and three retrieval perspectives. The existing DGE→BERT experiment does not clearly establish whether the augmented candidate pool, training supervision, and model capacity remain identical, so it cannot independently quantify the benefit of the derivation graph topology. Appendix Table 7 supports the effectiveness of the DGE retriever as a whole, but it likewise does not establish whether the gains primarily come from the graph structure.

This issue directly affects the assessment of the paper’s contribution. Under the same augmented data and generation pipeline, DGE should be compared with a trainable retriever without graph-neighbor interactions. Removing or shuffling derivation edges would further test whether the derivation relationships themselves provide additional value. The existing ablations are mainly conducted on a subset of approximately 300 examples, without reporting variability across independent runs, which also limits the interpretation of smaller gains.

  1. The accuracy–cost trade-off is a central selling point, but the current cost accounting and comparison settings do not sufficiently support the paper’s strong claims. The authors explicitly state that the main baselines use the models and settings from their respective papers, and that some costs are estimated. Such comparisons help position the system in practice, but make it difficult to distinguish cost differences attributable to the retrieval method, underlying model, number of generations, and inference configuration. Although Appendix Table 7 controls the generation pipeline, it does not also report token consumption, call counts, and latency for each retrieval strategy.

Moreover, the module-level statistics in Table 3 contain a clear inconsistency: the average output of the Information Extraction module alone is 0.36K tokens, already exceeding the 0.33K reported for the complete system, while the outputs of the online modules sum to approximately 0.64K. If these rows use different accounting conventions, those conventions should be explicitly explained; if this is a reporting error, the aggregate cost should be corrected. Offline question generation costs are also reported only partially, without a complete accounting of offline costs or an amortization analysis. Thus, the system appears to offer good online efficiency, but claims of the “lowest cost” and broad Pareto superiority require more consistent evidence.

Questions for authors

The following questions are intended to clarify the current conclusions, and the answers may change my assessment.

  1. Core contribution attribution: When the augmented candidate pool, training supervision, model capacity, and downstream generation pipeline are held constant, how much does performance decrease if graph-neighbor interactions are removed or derivation edges are shuffled? Which conditions are held constant in the existing DGE→BERT comparison and Table 7?

  2. Cost accounting: Please explain why Table 3 reports 0.36K output tokens for Information Extraction but only 0.33K for the complete system. Does the total include all three generation paths, all correction calls, and final selection? Can corrected module-level statistics be reconciled with the reported total?

  3. Controlled experiments and stability: Which dataset and evaluation subset are used in Table 7? Under this fixed pipeline, could you report each retriever’s accuracy, input and output token consumption, and variability across repeated runs, to establish whether retrieval gains and efficiency advantages hold simultaneously?

  4. Reproducibility of graph construction: Does node identity include the database identifier and normalized SQL? The pseudocode in Appendix C.1 creates and looks up nodes using question–SQL labels, whereas the text states that nodes are uniquely identified by SQL. How are these two descriptions reconciled? Also, how exactly is the criterion in Table 4 that “the cumulative execution of subqueries is equivalent to the original query” defined and implemented?

1 comment

Thank you for the constructive feedback!

Q1 & W1:

To examine the contribution of graph structure, we conducted edge-ablation experiments on the full BIRD-Dev set using DeepSeek-V3. Across the three methods, we kept the augmented candidate pool, training supervision, model capacity, and downstream generation pipeline unchanged. In the random-edge variant, each seed node connects to 10 randomly sampled nodes, using the maximum subgraph-size parameter.

MethodsEX
Derivation-SQL66.62
No edges63.95
Random edges64.93

Removing and randomizing the edges reduce EX by 2.67% and 1.69%, respectively. These comparisons support the additional benefit of the original derivation connections under the same augmentation and training conditions.

We also extend the ablation analysis to the full BIRD-Dev set using DeepSeek-V3:

MethodsEX
Derivation-SQL66.62
w/o Text-based retrieval64.6
w/o SQL-based retrieval65.38
w/o Hybrid retrieval64.02
DGE -> Bert64.21
DGE -> Bert*64.8
only training set64.67
only training set*65.25
only Text-based retrieval*65.19
only SQL-based retrieval*64.54
only Hybrid retrieval*65.78

The original DGE->BERT comparison and Table 7 retain the downstream generation settings of Derivation-SQL, with the example retriever replaced. In particular, Table 7 fixes the LLM, prompt, number of retrieved examples, and downstream pipeline, as described in Appendix E.8. These are retriever-level comparisons; the edge ablations above more directly examine graph structure while controlling the augmented candidate pool and model capacity.

Q2 & W2:

We apologize for the reporting error: the Information Extraction output in Table 3 should be 0.036K, not 0.36K.

Offline question generation runs once and is reused during inference. Using the reported per-question averages, 15,641 questions require approximately 29.4 million input tokens and 1.4 million output tokens. These are estimates from rounded averages. The total cost for one query by Derivation-SQL is online cost + offline cost / Q, where offline cost includes question generation, DGE training, and decomposition, and Q is the number of queries served without rebuilding. Applying our accounting formula to the corrected module-level statistics yields the 0.33K whole-system output total. As online Text-to-SQL calls increase, the offline cost per call becomes negligible.

Q3 & W2:

Table 7 uses BIRD-SDS, the 301-question BIRD-Dev subset described in Appendix E.1. For GPT-4o, we summarize the EX results from Table 7 alongside the additional token and latency statistics:

MetricsDGERandomSemanticSQL-structureAST-based
EX (%)73.0965.4564.7868.1163.79
Tokens (k)11.486.586.538.346.64
Latency (s)7.456.137.397.227.39

DGE achieves the highest EX among all five retrievers. Compared with SQL-structure, the strongest alternative, it improves EX by 4.98%, while measured latency increases from 7.22 s to 7.45 s. With 3.14K additional tokens per query, DGE provides higher SQL accuracy with a 0.23 s increase in response time, highlighting its value for accuracy-sensitive applications. DGE uses three groups of retrieved examples to generate three SQL candidates for final selection.

Q4:

Nodes are keyed by exact SQL strings, without database identifiers or normalized SQL. The question–SQL pairs in Algorithm 1 are node labels. Each unique SQL identifies one node with exactly one associated question: an original dataset SQL keeps its human question, while only a decomposed SQL without one receives a generated question. For Derivation-SQL, we consider the derived SQL queries valid and executable, indicating that the cumulative execution of subqueries is equivalent to the original query. Because the decomposition algorithm iteratively removes operations of a SQL query, the derived SQL queries are the parts of the SQL query. For example, the SQL query T “SELECT * FROM A INNER JOIN B ON A.ID = B.ID” could derive “SELECT * FROM A” A and “SELECT * FROM B” B, the executable result of T is equal to the cumulative execution of A, B and T. Moreover, the decomposition algorithm is not a probabilistic model. When implemented correctly, it guarantees correctness under the defined extraction criteria. In contrast, probabilistic models can produce multiple derived SLQ queries, and some of them may violate the extraction criteria even with correct implementation. DG uses these structural derivations to retrieve related SQL examples, whose utility is supported by the edge ablations in Q1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.