- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes RSTAR, a closed-loop Text-to-SQL framework with three modules: a topology-aware schema linker, a retriever, and an execution-feedback refinement agent. Experiments on multiple benchmarks demonstrate that RSTAR achieves strong results and surpasses prior state-of-the-art methods.
Critical strengths
S1. The paper identifies three concrete failure modes in enterprise Text-to-SQL (schema topology ambiguity, retrieval mismatch, execution-level blind spots) and designs dedicated modules for each.
Critical weaknesses
W1. The topological waveform fingerprint is presented as the key innovation of the retriever, yet the paper fails to compare it against standard retrieval baselines such as BM25 or dense retrieval (e.g., Sentence-BERT). Without these comparisons, the claimed superiority of the fingerprint-based retrieval is unsubstantiated.
W2. The RGAT encoder's training objective and supervision signals are underspecified. Moreover, no ablation study replaces RGAT with simpler alternatives (GCN, RGCN, vanilla GAT) to isolate its contribution. Combined with the questionable practice of using raw cross-attention scores as fingerprints without calibration, it is difficult to determine whether the reported gains stem from the graph encoder, the fingerprint design, or other components.
Questions for authors
Q1. Did you compare your topological fingerprint retrieval against BM25 or dense retrieval? If not, how do you rule out that a simple baseline would match or exceed your results?
Q2. What is the exact supervision signal for RGAT training (e.g., node classification, link prediction, contrastive learning)? How are positive and negative pairs defined, and is RGAT trained end-to-end with the retriever?
Q3. Raw cross-attention scores are known to be unstable and may not reflect true relevance. Did you experiment with any calibration (e.g., through contastive learning) before forming the fingerprint? An ablation study comparing raw attention fingerprints against calibrated ones would directly address this concern.
Response to Reviewer PZnX
Dear Reviewer PZnX,
Thank you for the precise review. We ran all requested experiments and report the results as they are, including where they qualify our claims. Table/section numbers refer to the revised PDF.
W1/Q1: Comparison vs. BM25 / dense retrieval (retriever ladder)
Spider dev (1,034) + BIRD dev (1,534), XiYanSQL-14B, identical downstream pipeline. Recall@k hit = SQL-skeleton match with gold (skeletonization: App. A.2.2); EX also reported.
Spider: each component contributes as claimed—RGAT + skeleton contrastive lifts R@5 35.7→47.8, the fingerprint adds +3.1; the full retriever is best on every metric. EX gaps (≤1.0) are within noise: the linker/generator compensate for retrieval noise, so Recall@k is the informative metric.
BIRD: the ordering inverts; we investigated rather than papering over it. Our fingerprint spans a fixed 10-table×20-column window, and 54.5% of BIRD databases (60.8% of questions) exceed it (Table 7)—avg 70 columns/DB, 231 columns + FK edges truncated—vs 13.9% on Spider. The topology the RGAT/fingerprint rely on is largely absent at input time on BIRD; the degradation is caused by the window, not the topology-aware principle. Implications: (1) our Table 1 BIRD result (76.66% EX) came from this same windowed retriever—it is a lower bound; (2) adaptive/hierarchical windows are now explicit future work (Sec. 5). Full ladder + coverage stats: App. A.5.2 (Table 8); protocol in A.5.1.
W2/Q2: What supervises the RGAT?
Apologies for the ambiguity—no node/link-prediction labels; supervision is entirely at the representation level, in two phases (implementation-verified):
GNN-variant ablation (new, single runs, identical budget):
On Spider RGAT is competitive with the best variant within training variance (6K samples, batch 8); on BIRD all graph variants degrade and GCN's parameter-free smoothing is most robust—consistent with truncation, since relation-specific attention cannot manifest when half the schemas lose FK edges. Hyperparameters (LR, τ) were tuned with RGAT (Fig. 12), not re-tuned per variant. Claims softened: the revision states "topology-aware schema encoding" with RGAT as our instantiation (App. A.5.3, Table 9).
W3/Q3: Calibration — Raw vs. Calibrated
Correct: the submitted fingerprint uses raw head-averaged weights. We renormalize with softmax(logits/τ) before sort-and-concatenate and swept τ ∈ {0.25, 0.5, 1.0, 2.0, 4.0}:
Clear gain where the fingerprint is informative (Spider: +3.7 R@3, +1.3 EX); neutral where the window truncates (BIRD, within noise). We adopt τ*=0.25 for both datasets (App. A.5.6, Table 12). Raw attention is indeed miscalibrated; the fix is one dev-tuned scalar and improves the deployed metric.
Summary of changes: (i) new App. A.5 (Tables 7–12) under an explicit protocol; (ii) Sec. 3.3/A.2.2 rewritten with the exact Phase-I/II objectives; (iii) RGAT claims scoped; coverage statistics motivate window-adaptive fingerprints as future work. Your scrutiny led to what we consider the revision's most interesting analysis—thank you.