- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This article first uses questions and database schema to obtain top M question examples via SchemaRAG. Then, for candidate SQL statements, numeric values and literals are replaced with placeholders. Next, an encoder scores R candidate SQLs using cosine similarity. By encoding, the specific literal values in the candidate SQLs are removed, and the cosine similarity between each candidate SQL and the query is computed, resulting in scores for the R candidates. Finally, the candidates are ranked, and the top K among them are selected. The LLM then generates executable SQL based on the target question, schema context, and the K reference examples.
Critical strengths
- SARank find "demonstration exposure bottleneck" problem and solve that by improving BestSim@3 from 0.714 to 0.834 and improving Recall@3 from 0.335 to 0.541.
- SARank improves over SchemaRAG in all 12 reported model–metric combinations across Spider and BIRD. The method they used, which are the masking and negative-sampling ablations, strengthens my assessment because the benefits extend beyond structural similarity proxies to SQL execution accuracy across the tested models and datasets.
Critical weaknesses
- The reranker’s training data, actual evaluation splits and sample counts, and demonstration-bank isolation rules are insufficiently specified, limiting reproducibility and assessment of evaluation independence.
- The evaluation lacks an untuned matched-backbone baseline and controlled comparisons with closely related structure-aware selection methods. Table 3 reports only structural metrics, leaving the downstream benefit of mixed-negative training unclear.
- Equation (5) does not specify the structural features or how they encode nesting, scope, and repeated operators. This makes it difficult to determine whether BestSim and Recall capture the structural alignment claimed by the paper.
Questions for authors
- Please specify the training and checkpoint-selection data, actual evaluation splits and sample counts, demonstration-bank sources, and rules excluding evaluation examples and their perturbations. Are reported accuracies from single runs or averages?
- Does the SchemaRAG row represent the original complete system or the shared generation pipeline with original SAR ordering? Were schema-linking outputs, decoding settings, and SQL refinement and final-selection stages held fixed?
- Could you add an untuned Qwen3-Embedding-0.6B baseline with identical masking, scoring, and candidate pools; report downstream EX for the Table 3 variants; and compare against a closely related structure-aware selector under controlled settings?
- Please define the feature extraction procedure in Equation (5), including its treatment of SQL hierarchy and operator multiplicity. Could you validate the findings with a hierarchy-aware metric and examine their relationship with EX improvements?
3 comments
Thank you for the detailed responses and additional experimental results. I have no further questions at this stage.
Thank you very much for taking your valuable time to review our manuscript. We have carefully revised it point by point in accordance with your previous comments and have explained the corresponding changes in our response letter. If you consider that the revised manuscript has improved significantly, we would be grateful if you could reconsider your recommendation for this manuscript in light of the revised version during your subsequent evaluation.
Regardless of the outcome, we sincerely appreciate your professional comments and assistance.
Q1. Training and Evaluation Splits, Demonstration Bank Isolation, and Number of Runs Thank you for pointing out the insufficient description of our experimental protocol. We combine 8,659 training examples from Spider and 9,428 from BIRD, yielding a total of 18,087 examples to train a single reranker. We use the final checkpoint (checkpoint-3040) after four training epochs, without selecting checkpoints based on evaluation-set performance. The detailed evaluation splits have been added to Table 9 (Dataset Split and Evaluation Statistics) in Appendix C.7. For Dr.Spider, we evaluate on three selected perturbation subsets: NLQ_keyword_synonym, SQL_comparison, and DB_schema_synonym. The Spider and BIRD demonstration banks are constructed exclusively from their respective training data. For BEAVER, the dw_real split used to construct the demonstration bank is excluded from evaluation. We found no exact question–SQL pair overlaps between the evaluation sets and their corresponding demonstration banks for BEAVER or Dr.Spider. The main tables report the average results of three generation runs using the same checkpoint. Separately, Appendix C.4 reports the mean and standard deviation across three reranker training seeds.
Q2. Interpretation of the SchemaRAG Baseline and Controlled Conditions Thank you for highlighting this ambiguity. We have clarified in the experimental setup, the caption of Table 1, and the appendix describing the prompt template that the SchemaRAG baseline uses the original SAR ranking. SchemaRAG and SARank share the same Top-10 candidate pool, schema-linking outputs, prompt template, and downstream LLM, with each method selecting its Top-3 demonstrations according to its respective ranking. This clarification distinguishes the controlled comparison between SchemaRAG and SARank from the full-system comparisons with other methods.
Q3. Untuned Baseline, Downstream EX for Negative Sampling, and Comparison with a Structure-Aware Selector Following the reviewer's suggestions, we have added three sets of results: Table 2: We introduce an untuned Qwen3-Embedding-0.6B baseline, which achieves BestSim@3 / Recall@3 scores of 0.734 / 0.413, compared with 0.834 / 0.541 for SARank. Table 3: We report downstream EX on Spider and BIRD for all negative-sampling variants. Mixed negative sampling achieves 90.18% / 69.74% EX, outperforming the best single-source strategy by 1.95 / 1.44 percentage points, respectively. Appendix C.2: Under the same Top-10 candidate pool, Top-3 demonstration budget, and DeepSeek-V4-Flash generation pipeline, SARank outperforms the ASTReS ranking component by 2.23 / 2.63 EX percentage points on Spider and BIRD, respectively. Together, these results provide additional evidence from an untuned baseline using the same backbone, the downstream effects of different training strategies, and a direct controlled comparison with a related structure-aware demonstration selector.
Q4. Structural Features, Operator Multiplicity, and Hierarchical Information Thank you for the suggestion. We have clarified the structural feature definitions around Eq. (5) and in Appendix H. After SQL normalization and masking string and numeric literals, we extract binary presence features for SQL clauses, aggregation functions, set operations, and window operations. We also record whether nesting is present and encode the occurrence counts of joins, subqueries, predicates, conjunctions, and disjunctions into four bins: 0, 1, 2, and 3+. Thus, the current metric captures coarse-grained operator multiplicity. The additional downstream EX results and demonstration utility analysis in Appendix C.3 provide complementary evidence.