acceptodds
Under review as a conference paper at ICLR 2027

SARank: Structure-Aware Demonstration Reranking for Text-to-SQL

Abstract

Text-to-SQL translates natural-language questions into executable SQL queries. Existing retrieval-augmented methods usually rank demonstrations by semantic or schema similarity. However, semantically similar questions may require different SQL operations, causing structurally relevant demonstrations to fall outside the limited Top- context. We call this problem the demonstration exposure bottleneck. To address it, we propose SARank, a structure-aware demonstration reranking method. SARank first uses a schema-aware retriever to construct a high-recall candidate pool. It then abstracts SQL literals and learns question–SQL compatibility with a shared encoder trained by a candidate-aware contrastive objective using retrieval-based, same-database, and random negatives. The resulting scores reorder the candidates before prompt construction, without requiring a predicted target SQL or execution feedback. Experiments across Spider, BIRD, BEAVER, and Dr.Spider, including evaluations with multiple LLMs, show that SARank improves , , and downstream execution accuracy. Candidate-budget analysis and case studies further show that SARank promotes lower-ranked but structurally aligned demonstrations into the Top-, providing more effective guidance for SQL generation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Text-to-SQL, Large Language Model, Contrastive Learning, In-Context Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_giTVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article first uses questions and database schema to obtain top M question examples via SchemaRAG. Then, for candidate SQL statements, numeric values and literals are replaced with placeholders. Next, an encoder scores R candidate SQLs using cosine similarity. By encoding, the specific literal values in the candidate SQLs are removed, and the cosine similarity between each candidate SQL and the query is computed, resulting in scores for the R candidates. Finally, the candidates are ranked, and the top K among them are selected. The LLM then generates executable SQL based on the target question, schema context, and the K reference examples.

Critical strengths
  1. SARank find "demonstration exposure bottleneck" problem and solve that by improving BestSim@3 from 0.714 to 0.834 and improving Recall@3 from 0.335 to 0.541.
  2. SARank improves over SchemaRAG in all 12 reported model–metric combinations across Spider and BIRD. The method they used, which are the masking and negative-sampling ablations, strengthens my assessment because the benefits extend beyond structural similarity proxies to SQL execution accuracy across the tested models and datasets.
Critical weaknesses
  1. The reranker’s training data, actual evaluation splits and sample counts, and demonstration-bank isolation rules are insufficiently specified, limiting reproducibility and assessment of evaluation independence.
  2. The evaluation lacks an untuned matched-backbone baseline and controlled comparisons with closely related structure-aware selection methods. Table 3 reports only structural metrics, leaving the downstream benefit of mixed-negative training unclear.
  3. Equation (5) does not specify the structural features or how they encode nesting, scope, and repeated operators. This makes it difficult to determine whether BestSim and Recall capture the structural alignment claimed by the paper.
Questions for authors
  1. Please specify the training and checkpoint-selection data, actual evaluation splits and sample counts, demonstration-bank sources, and rules excluding evaluation examples and their perturbations. Are reported accuracies from single runs or averages?
  2. Does the SchemaRAG row represent the original complete system or the shared generation pipeline with original SAR ordering? Were schema-linking outputs, decoding settings, and SQL refinement and final-selection stages held fixed?
  3. Could you add an untuned Qwen3-Embedding-0.6B baseline with identical masking, scoring, and candidate pools; report downstream EX for the Table 3 variants; and compare against a closely related structure-aware selector under controlled settings?
  4. Please define the feature extraction procedure in Equation (5), including its treatment of SQL hierarchy and operator multiplicity. Could you validate the findings with a hierarchy-aware metric and examine their relationship with EX improvements?
3 comments

Q1. Training and Evaluation Splits, Demonstration Bank Isolation, and Number of Runs Thank you for pointing out the insufficient description of our experimental protocol. We combine 8,659 training examples from Spider and 9,428 from BIRD, yielding a total of 18,087 examples to train a single reranker. We use the final checkpoint (checkpoint-3040) after four training epochs, without selecting checkpoints based on evaluation-set performance. The detailed evaluation splits have been added to Table 9 (Dataset Split and Evaluation Statistics) in Appendix C.7. For Dr.Spider, we evaluate on three selected perturbation subsets: NLQ_keyword_synonym, SQL_comparison, and DB_schema_synonym. The Spider and BIRD demonstration banks are constructed exclusively from their respective training data. For BEAVER, the dw_real split used to construct the demonstration bank is excluded from evaluation. We found no exact question–SQL pair overlaps between the evaluation sets and their corresponding demonstration banks for BEAVER or Dr.Spider. The main tables report the average results of three generation runs using the same checkpoint. Separately, Appendix C.4 reports the mean and standard deviation across three reranker training seeds.

Q2. Interpretation of the SchemaRAG Baseline and Controlled Conditions Thank you for highlighting this ambiguity. We have clarified in the experimental setup, the caption of Table 1, and the appendix describing the prompt template that the SchemaRAG baseline uses the original SAR ranking. SchemaRAG and SARank share the same Top-10 candidate pool, schema-linking outputs, prompt template, and downstream LLM, with each method selecting its Top-3 demonstrations according to its respective ranking. This clarification distinguishes the controlled comparison between SchemaRAG and SARank from the full-system comparisons with other methods.

Q3. Untuned Baseline, Downstream EX for Negative Sampling, and Comparison with a Structure-Aware Selector Following the reviewer's suggestions, we have added three sets of results: Table 2: We introduce an untuned Qwen3-Embedding-0.6B baseline, which achieves BestSim@3 / Recall@3 scores of 0.734 / 0.413, compared with 0.834 / 0.541 for SARank. Table 3: We report downstream EX on Spider and BIRD for all negative-sampling variants. Mixed negative sampling achieves 90.18% / 69.74% EX, outperforming the best single-source strategy by 1.95 / 1.44 percentage points, respectively. Appendix C.2: Under the same Top-10 candidate pool, Top-3 demonstration budget, and DeepSeek-V4-Flash generation pipeline, SARank outperforms the ASTReS ranking component by 2.23 / 2.63 EX percentage points on Spider and BIRD, respectively. Together, these results provide additional evidence from an untuned baseline using the same backbone, the downstream effects of different training strategies, and a direct controlled comparison with a related structure-aware demonstration selector.

Q4. Structural Features, Operator Multiplicity, and Hierarchical Information Thank you for the suggestion. We have clarified the structural feature definitions around Eq. (5) and in Appendix H. After SQL normalization and masking string and numeric literals, we extract binary presence features for SQL clauses, aggregation functions, set operations, and window operations. We also record whether nesting is present and encode the occurrence counts of joins, subqueries, predicates, conjunctions, and disjunctions into four bins: 0, 1, 2, and 3+. Thus, the current metric captures coarse-grained operator multiplicity. The additional downstream EX results and demonstration utility analysis in Appendix C.3 provide complementary evidence.

Thank you for the detailed responses and additional experimental results. I have no further questions at this stage.

Thank you very much for taking your valuable time to review our manuscript. We have carefully revised it point by point in accordance with your previous comments and have explained the corresponding changes in our response letter. If you consider that the revised manuscript has improved significantly, we would be grateful if you could reconsider your recommendation for this manuscript in light of the revised version during your subsequent evaluation.

Regardless of the outcome, we sincerely appreciate your professional comments and assistance.

Reviewer_v7zbOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the demonstration exposure bottleneck in Text-to-SQL: a useful SQL pattern may already be in the retrieved candidate pool but fall outside a small Top-K prompt.

SARank takes the Top-10 examples retrieved by SchemaRAG, masks SQL literals, and uses a shared encoder to rerank candidate SQL queries against the target question. The encoder starts from Qwen3-Embedding-0.6B and is trained with paired SQL positives and retrieved, same-database, and random negatives. The top three candidates are then used as demonstrations.

Critical strengths
  1. The paper addresses a concrete problem in few-shot Text-to-SQL: structurally relevant examples may already be retrieved but excluded from the Top-K demonstrations. SARank directly ranks candidate SQL queries against the input question without requiring a predicted target SQL. This makes it straightforward to integrate into existing retrieval-based pipelines.

  2. The authors keep the candidate pool, prompt, and downstream LLM fixed when evaluating reranking. SARank improves execution accuracy over SchemaRAG across three LLMs on both Spider and BIRD (Table 1). These results provide evidence that better demonstration selection can improve SQL generation without modifying the underlying generator.

Critical weaknesses
  1. The methodological novelty is somewhat limited. SARank combines a pretrained bi-encoder, SQL literal masking, and contrastive learning with mixed negatives. Although the ablations support these design choices, it remains unclear how much improvement comes from task-specific fine-tuning of the strong pretrained encoder. A comparison with the unfine-tuned Qwen3-Embedding-0.6B under the same candidate pool would help clarify the incremental contribution. It remains unclear how much of the improvement comes from the pretrained encoder, standard question–SQL contrastive learning, or the proposed negative mixture. This is important for assessing the technical contribution.

  2. The structural evaluation is relatively coarse. The Jaccard-based metric in Eq. (5) cannot distinguish SQL queries with similar operator sets but different nesting or join structures. Although the downstream execution results provide additional evidence, some improvements are modest (e.g., +0.72 EX points on BIRD with GPT-4o), and no statistical significance tests or variance estimates are reported. This makes the reliability of the smaller gains difficult to assess.

Questions for authors
  1. How does the unfine-tuned Qwen3-Embedding-0.6B perform on the same SchemaRAG candidate pool? This comparison would help separate the benefit of the pretrained encoder from that of task-specific fine-tuning.

  2. Can the authors evaluate structural similarity using a representation that captures SQL nesting and operation composition, and test whether the gains persist when schema identifiers are abstracted? Please report paired significance tests or repeated runs for the smaller EX improvements.

  3. What latency and memory overhead does the reranker add per question? Are candidate SQL embeddings precomputed? Appendix C.2 reports V100 GPUs, whereas Appendix D.1 specifies BF16 training; since V100 does not support native BF16, please clarify the actual precision and hardware setup.

  4. Which source data were used to train the reranker for the zero-shot BEAVER and Dr.Spider evaluations, and how were their candidate demonstration banks constructed? This is needed to interpret the transfer results.

4 comments

Q1. Comparison with the Untuned Encoder Thank you for the suggestion. We have added an untuned Qwen3-Embedding-0.6B baseline to Table 2. This baseline uses the same SchemaRAG Top-10 candidate pool and evaluation pipeline as SARank, eliminating differences in candidate retrieval as a confounding factor. The untuned encoder achieves BestSim@3 and Recall@3 scores of 0.734 and 0.413, respectively, whereas SARank achieves 0.834 and 0.541, corresponding to absolute improvements of 0.100 and 0.128. These results support the effectiveness of task-specific fine-tuning compared with directly using the pretrained encoder. The baseline results and corresponding analysis are included in Table 2 of Section 4.4, “Controlled Comparison of Ranking Strategies.” In addition, Figure 5 presents execution accuracy across BestSim@3 intervals to examine the relationship between structural matching quality and downstream execution accuracy.

Q2. Structural Similarity and Reliability of Experimental Results Thank you for your attention to the structural evaluation metric and experimental reliability. First, Appendix H provides additional details on structural feature extraction and similarity computation. The metric includes SQL nesting indicators and operator-count bins, but does not explicitly encode complete AST hierarchies. We acknowledge this limitation and will state it explicitly in the revised manuscript. Second, Appendix C.4 reports experiments with three reranker training seeds: 42, 2024, and 2025. The mean EX scores and sample standard deviations across these seeds are 90.42 ± 0.70% on Spider and 69.83 ± 0.24% on BIRD. All three seeds outperform the corresponding fixed baseline on both datasets, providing evidence of consistency under the evaluated settings.

Q3. Computational Overhead, Candidate Embedding Caching, and Training Configuration Thank you for your attention to computational efficiency and implementation details. We provide the following clarification on reranking time, candidate embedding caching, memory requirements, and the training configuration. (1)Reranking time. In batch evaluation, processing 1,034 Spider development questions takes 48.44 seconds, while processing 1,534 BIRD development questions takes 81.99 seconds. These correspond to approximately 46.85 ms and 53.45 ms per question, respectively. The measurements include SQL literal masking, input preparation, encoding of the question and its Top-10 candidate SQL queries, similarity computation, and sorting. (2) Candidate embedding caching. The current implementation does not use a persistent cache of precomputed candidate SQL embeddings. Questions and candidate SQL queries are encoded during each evaluation run, so the reported times include candidate encoding costs. For a fixed demonstration bank, precomputing and caching candidate embeddings could avoid redundant encoding. However, we have not separately measured the performance benefits of this optimization. (3) Memory requirements. The reranker contains approximately 0.6 billion parameters, requiring about 1.2 GB for model weights stored in FP16. Assuming FP16 storage, the 1,024-dimensional embeddings for one question and ten candidate SQL queries require approximately 22 KiB. These are theoretical storage estimates, excluding intermediate activations, temporary tensors, and framework overhead. (4) Training hardware and precision. After checking the training configuration, we confirm that training used NVIDIA V100 GPUs with FP16 mixed precision. BF16 was incorrectly reported in the original manuscript due to a configuration documentation error; this did not affect the actual training procedure or experimental results. We will replace the outdated statement about the absence of latency measurements in Section 3.4, “Selection Within a Fixed Candidate Pool,” with the measured times and their scope and limitations. We will also add the timing scope, caching details, and memory estimates to Appendix C.5, “Compute and Software.” The training precision has been corrected to FP16 in Table 10 of Appendix D.1, “Implementation Details.”

Q4. Data Sources for Zero-Shot Transfer Evaluation on BEAVER and Dr.Spider Thank you for your attention to the transfer evaluation setup and data separation. For both BEAVER and Dr.Spider, we use the same reranker trained on the Spider and BIRD training sets, without additional fine-tuning on either target dataset. For BEAVER, we construct the candidate demonstration bank from 121 examples in the dw_real split. This split is excluded from evaluation, which is conducted only on dw, nova, and neutron. For Dr.Spider, we reuse the candidate bank of 3,102 demonstrations constructed from Spider training data. We evaluate on three selected perturbation subsets: NLQ_keyword_synonym, SQL_comparison, and DB_schema_synonym. Our check found no exact natural-language question–SQL pair matches between the respective evaluation sets and demonstration banks. This check rules out direct duplication of those pairs, rather than all possible forms of overlap. We further clarify that “zero-shot transfer” refers to applying the reranker without additional parameter fine-tuning on the target dataset; it does not mean that inference uses no demonstrations. We will incorporate these data sources and evaluation settings explicitly into the revised manuscript.

Thank you very much for taking your valuable time to review our manuscript. We have carefully revised it point by point in accordance with your previous comments and have explained the corresponding changes in our response letter. If you consider that the revised manuscript has improved significantly, we would be grateful if you could reconsider your recommendation for this manuscript in light of the revised version during your subsequent evaluation.

Regardless of the outcome, we sincerely appreciate your professional comments and assistance.

Thank you for the detailed response and additional experiments.

There are two points still expected: (1) please also report Execution Accuracy for the untuned encoder baseline, not just BestSim@3/Recall@3; (2) the similarity metric still does not explicitly model SQL nesting and operator composition. I suggest noting this limitation and possible extensions (e.g., AST/tree-edit distance) as future work.

Overall, the rebuttal has addressed most of my concerns, and I will maintain my original rating.

Reviewer_DJCaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SARank, a structure-aware demonstration reranking framework for retrieval-augmented Text-to-SQL generation. The method addresses the demonstration exposure bottleneck, where structurally relevant examples are retrieved but ranked below the limited Top-K demonstration budget. SARank first retrieves a fixed candidate pool using a schema-aware retriever and then reranks the candidates based on learned question–SQL compatibility. The reranker employs SQL literal masking and a shared encoder trained with a contrastive objective incorporating retrieval-based hard negatives, same-database negatives, and random negatives. Experiments on Spider, BIRD, BEAVER, and Dr.Spider demonstrate improvements in structural retrieval metrics and downstream execution accuracy across multiple language models. Ablation studies further investigate the effects of literal masking, negative sampling, and demonstration budgets.

Critical strengths
  1. The paper identifies a practical limitation of retrieval-augmented Text-to-SQL systems: semantically relevant demonstrations may fail to expose the SQL structures needed for the target query.
  2. SARank reranks a fixed candidate pool while keeping the downstream prompt and generation model unchanged, helping isolate the contribution of demonstration ordering.
  3. The method improves structural retrieval metrics and execution accuracy on Spider and BIRD across three language models, with additional transfer results on BEAVER and Dr.Spider.
Critical weaknesses
  1. This paper's distinction from existing structure-aware demonstration selection methods is not sufficiently clear.
  2. Direct controlled comparisons with structure-aware demonstration selectors, particularly ASTReS, are missing.
  3. The contrastive objective distinguishes paired gold SQL from unpaired queries, but structural compatibility does not necessarily reflect how much a demonstration improves downstream generation.
Questions for authors
  1. Could the authors clarify the main technical differences between SARank and existing structure-aware demonstration selection methods, especially ASTReS?
  2. The contrastive objective treats the paired gold SQL as positive and unpaired SQL queries as negatives. However, an unpaired query may still be a useful demonstration. Could the authors analyze the correlation between the learned reranking scores and the actual downstream benefit of individual demonstrations?
  3. Could the authors report results across multiple reranker training seeds and provide paired significance tests or confidence intervals for the execution-accuracy improvements on Spider and BIRD?

If the authors can address these issues, I will consider raising the score.

2 comments

Q1. Technical Differences from ASTReS and Controlled Comparison Thank you for the suggestion. We have clarified the technical distinction from ASTReS and included a controlled comparison in Appendix C.2. The main distinction is that SARank reranks demonstrations without first predicting the target SQL. ASTReS uses the AST of an approximated target SQL as a ranking reference, so its structural matching can be affected by errors in that intermediate prediction. SARank instead learns compatibility directly between the question and each candidate SQL. We train this scorer using retrieval-based hard negatives, same-database negatives, and random negatives to distinguish relevant-looking candidates with different SQL operation requirements. In the controlled comparison, all three ranking methods use the same Top-10 candidate pool, select three demonstrations, and share the DeepSeek-V4-Flash generation pipeline and prompt template. Original SAR achieves EX scores of 88.90% on Spider and 67.02% on BIRD; the ASTReS ranking component achieves 84.31% and 62.52%; and SARank achieves 90.18% and 69.74%. SARank therefore improves over the ASTReS ranking component by 5.87 and 7.22 percentage points, respectively.

Q2. Learned Ranking Scores and the Downstream Utility of Individual Demonstrations Thank you for highlighting the distinction between paired-SQL supervision and demonstration utility. We agree that an SQL query not paired with the target question may still be a useful demonstration. Accordingly, we clarify that our contrastive objective is a proxy for demonstration selection: a negative label denotes an unpaired SQL query, not a demonstration that is necessarily unhelpful for generation. Appendix C.3 analyzes individual demonstration utility on Spider by grouping candidates according to their within-question SARank positions: ranks 1–3, 4–7, and 8–10. For each group, incorrect-to-correct and correct-to-incorrect transitions are measured relative to the baseline, using all candidate evaluations in that group as the denominator. For ranks 1–3, the two transition rates are 25.60% and 7.20%, yielding a net EX gain of 18.40 percentage points. For ranks 4–7, they are 34.90% and 23.50%, yielding 11.40 points. For ranks 8–10, they are 48.30% and 47.80%, yielding 0.50 points. Thus, higher-ranked groups show larger net gains, primarily through fewer correct-to-incorrect changes rather than higher correction rates. Lower-ranked candidates can still help individual queries, but their improvements and degradations nearly cancel out in aggregate.

Q3. Reliability of Results Across Multiple Training Seeds Thank you for requesting additional reliability analysis. Appendix C.4 reports results from three reranker training seeds—42, 2024, and 2025—while reusing the same SchemaRAG baseline predictions for each dataset. On Spider, SARank achieves EX scores of 90.18%, 89.87%, and 91.20%, with a mean and sample standard deviation of 90.42 ± 0.70%, compared with 88.90% for SchemaRAG. On BIRD, the corresponding scores are 69.74%, 70.11%, and 69.65%, giving 69.83 ± 0.24%, compared with 67.02% for SchemaRAG. SARank outperforms the fixed baseline for all three training seeds. The improvements range from 0.97 to 2.30 percentage points on Spider and 2.63 to 3.09 points on BIRD, with mean improvements of 1.52 and 2.81 points, respectively. These results provide evidence that the observed gains are not limited to one reranker training seed.

Thank you very much for taking your valuable time to review our manuscript. We have carefully revised it point by point in accordance with your previous comments and have explained the corresponding changes in our response letter. If you consider that the revised manuscript has improved significantly, we would be grateful if you could reconsider your recommendation for this manuscript in light of the revised version during your subsequent evaluation.

Regardless of the outcome, we sincerely appreciate your professional comments and assistance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.