- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Derivation-SQL, a graph-based framework that leverages derivation relationships among SQL queries for few-shot example retrieval in Text-to-SQL. The method decomposes training SQL queries to expand the example pool and constructs a SQL Derivation Graph (DG) that connects examples through these relationships. A Derivation Graph Embedding Network (DGE) then learns representations that incorporate question semantics and SQL derivation relationships, enabling retrieval directly from natural-language questions without preliminary SQL generation. Experiments on BIRD and Spider report competitive execution accuracy, with reduced token usage and inference costs in the reported comparisons.
Critical strengths
Strength 1. Strong performance with efficient inference.
The method achieves competitive execution accuracy on BIRD and Spider, with a promising balance between accuracy and inference cost. Offline graph construction and example encoding enable retrieval without preliminary SQL generation. With GPT-4o, it reports 72.56% execution accuracy on BIRD-Dev at an average cost of $0.031 per query.
Strength 2. An interesting use of SQL decomposition for example retrieval.
The idea of decomposing training SQL queries to construct additional examples and model their derivation relationships is interesting. Rather than treating these examples independently, the method uses their relationships to guide representation learning, connecting example augmentation with graph-based retrieval in a coherent framework.
Critical weaknesses
Weakness 1. Limited Novelty and Insufficient Differentiation from Prior Work
- The problem formulation is not clearly distinguished from prior work. The limitations of semantic retrieval and the cost of preliminary SQL generation have already been explored. DCG-SQL [1] addresses similar challenges, while ReFSQL [2] also proposes structure-aware question representations. The differences from prior work should be more clearly articulated.
- The paper does not sufficiently explain which limitations of existing methods remain unresolved or why modeling SQL derivation relationships is necessary. A clearer discussion of ReFSQL and DCG-SQL would better establish its unique contributions.
- Prior approaches such as GAT [3] and RAT-SQL [4] have already explored incorporating graph structures or relational information into attention mechanisms. The paper should clarify whether its encoder introduces novel architectural components or, if it builds on existing techniques, better explain the motivation for adopting them in the context of SQL derivation graphs.
- The paper lacks direct comparisons with existing structure-aware retrieval methods that avoid preliminary SQL generation. Such comparisons are necessary to demonstrate the specific effectiveness and efficiency advantages of the proposed approach.
Weakness 2. Limited analysis of the methodology
- For DGE, the motivation for the hybrid encoder is insufficiently explained. Although lines 275–279 describe the encoder, the discussion focuses primarily on implementation details rather than explaining why a separate Hybrid encoder is necessary.
- In addition, the paper lacks a dedicated ablation study that isolates the contribution of DG. Although DG is presented as one of the paper’s core contributions, it remains unclear how much it contributes to the reported performance improvements. Further analysis is needed to determine how the extent of few-shot example augmentation through SQL decomposition affects downstream performance.
- Several design choices lack sufficient explanation, particularly the concatenation of mean-pooled representations from the last three BERT layers. Further discussion or ablation studies would help clarify why these specific choices were made and what benefits they provide.
[1] Lee, J., et al. DCG-SQL: Enhancing In-Context Learning for Text-to-SQL with Deep Contextual Schema Link Graph
[2] Zhang, K., et al. ReFSQL: A Retrieval-Augmentation Framework for Text-to-SQL Generation
[3] Veličković, P., et al. Graph Attention Networks. International Conference on Learning Representations
[4] Wang, B., et al. RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
Questions for authors
Question 1. Quality and diversity of augmented few-shot examples.
Although Appendix E.6 evaluates the quality of the generated questions, I would like to understand what additional structural diversity the augmented examples introduce and how much they contribute to downstream performance. Could the authors provide a quantitative analysis of this diversity, such as coverage of SQL patterns or operation combinations, and assess the performance gains attributable to example augmentation? A controlled comparison between the original, non-augmented candidate pool and the augmented pool, using the same retrieval method and downstream pipeline, would help clarify the effectiveness of the augmentation.
Question 2. Contribution of graph structure.
The proposed DG offers an interesting way to model SQL derivation relationships. However, I would like to understand how much additional benefit the graph structure provides beyond expanding the few-shot candidate pool. Could the authors isolate the contribution of graph connectivity by comparing the proposed method with a variant that uses the same augmented examples but without graph-based encoding, while keeping the remaining experimental settings fixed?
4 comments
R2
For Q1:
We supplement a diversity analysis of DG which including both the training set and the derived data. Values inside parentheses denote the proportion of samples within the corresponding data. We count the number of samples with 1, 2, 3, or (>= 4) JOIN clauses, as well as samples with 1, 2, 3, 4, or (>= 5) query predicates. Nested Queries and Aggregate Functions report the count of samples that include at least one nested query or aggregate function, respectively. Compared with the distribution of the training set, the number of each type SQL queries in the DG have significant improvement.
| Data | Size | JOIN (1/2/3/>=4) | Query Predicates (1/2/3/4/>=5) | Nested Queries | Aggregates Function |
|---|---|---|---|---|---|
| Training set | 9428 | 3038 (32.22) /4683 (49.67) /1355 (14.37)/352 (3.73) | 1212 (12.86)/1520 (16.12)/2839 (30.11)/2097 (22.24)/1760 (18.67) | 633 (6.71) | 4002 (42.45) |
| DG | 25069 | 8207 (32.74)/12046 (48.05)/3749 (14.95)/1067 (4.26) | 4072 (16.24)/4954 (19.76)/7465 (29.78)/4898 (19.54)/3680 (14.68) | 1138 (4.54) | 5416 (21.6) |
To quantify the performance gain brought by this augmentation, we refer to the controlled ablation in W2: under the same retrieval and LLM pipeline, the model with the augmented DG pool achieves 66.62% EX, while the non-augmented setting (w/o DG, using only the original training set as candidates) reaches 64.67% EX. This demonstrates that the structurally diverse augmented examples bring consistent performance improvements.
For Q2:
To isolate the contribution of graph connectivity, we conduct additional ablation experiments on BIRD-Dev. All variants share the same augmented candidate examples from DG; we only modify the derivation edges while keeping all other settings fixed.
| Methods | EX |
|---|---|
| Derivation-SQL | 66.62 |
| No edges | 63.95 |
| Random edges | 64.93 |
The results show that the graph structure with authentic derivation edges brings clear performance gains. Removing all derivation edges leads to a substantial EX drop. Randomly shuffled edges yield better performance than the edge-free variant but still underperform the original DG, which demonstrates that the true SQL derivation connectivity, rather than arbitrary graph connections, is essential for our method.
Thank you for your detailed response. I encourage the authors to incorporate the clarifications and additional experimental results into the revised paper. I will raise my score.
R2
Thank you for your careful review and for acknowledging our contributions. We will incorporate all these clarifications and additional experimental results into the revised paper.
R1
Thank you for the constructive feedback!
Considering cost and efficiency, we conduct the experiment using the Deepseek-v3 as the LLM backbone.
For W1:
1 & 2: We clarify that our work is different from DCG-SQL and ReFSQL. As discussed in the introduction and related work, Derivation-SQL leverages SQL decomposition to construct a larger pool of fine-grained few-shot examples for retrieval, which DCG-SQL and ReFSQL do not possess. These decomposed examples offer more granular guidance for LLMs. To retrieve few-shot examples more effectively, Derivation-SQL designs a novel retrieval method tailored to the characteristics of derivation examples, which enables retrieval beyond vanilla semantic matching. Derivation-SQL features a fundamentally different graph construction: it models derivation relationships among SQL queries, in contrast to DCG-SQL and ReFSQL, which prioritize schema relationships and token-level semantics between questions and database schemas.
Technically, there are several key differences. First, the graph construction is different. DCG-SQL and ReFSQL construct graphs based on question–schema and token semantic relations, while our derivation graph captures SQL-to-SQL derivation dependencies. Second, the encoding pipeline is different. Derivation-SQL is more efficient than baselines. DCG-SQL and ReFSQL need to use graph neural networks. In contrast, Derivation-SQL encodes nodes offline. During online inference, Derivation-SQL encodes the question without graph computation and uses dot-product similarity for node retrieval. Third, DCG-SQL and ReFSQL require few-shot examples and the target schema to belong to the same database, since their graph nodes for questions and schemas must interact. This constraint limits their applicability on benchmarks like BIRD, where training and dev databases are disjoint. Our method removes this restriction.
Notably, although DCG-SQL and ReFSQL introduce structure-aware representations, they do not model how SQL queries are derived from one another. This derivation signal is the core unresolved limitation we target: semantic or question-schema matching does not explicitly model the derivation paths between SQL variants, which are valuable for guiding complex SQL generation.
3: GAT is a general graph attention architecture and is not tailored for our derivation graph. RAT-SQL is a structure-aware method designed for question–schema modeling, which focuses on capturing schema connections and alignments between questions and database columns. In contrast, our work targets derivation relationships among SQL queries. Therefore, neither method can be directly adopted for our derivation graph.
4: We compare our DGE with the structure-enhanced retriever (SER) from ReFSQL. Following the setup of ReFSQL, we train BERT via contrastive learning to encode both questions and graph nodes. SQL queries with high tree edit distances are treated as positive samples, and others as negatives.
As shown in the table, the performance of DGE is better than that of SER under both GPT-4o and Gemini-2.5-flash.
For W2:
1: The motivation for the hybrid encoder is to integrate text-based information and SQL-based information, so as to provide a new aspect to retrieve examples. The hybrid encoder could help retrieve different examples, making the retrieval more robust.
2: To verify the effectiveness of the DG, we conduct experiments by using DG or not. When removing DG, we use the training set as the few-shot example provider and use Bert to encode the question of the training set.
From the table, we find that: First, directly using DG without DGE, the performance is weaker than that of using the original training set. This is because Bert cannot capture the structure information, while the similar expression of questions misleads BERT.
3: The concatenation of mean-pooled representations from the last three BERT layers is a common trick for improving performance. Because each layer of Bert captures different information, mean-pooled representations from the last three BERT could provide more information for subsequent module.