acceptodds
Withdrawn from ICLR 2027

RSTAR: Relational and Semantic Topology-Aware Refinement for Text-to-SQL

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-to-SQL aims to translate natural language questions into executable Structured Query Language (SQL) statements, allowing users to access relational databases through plain language instead of formal query syntax. Recent advances in Large Language Models (LLMs) have established prompt-based reasoning and retrieval-augmented generation as dominant paradigms for Text-to-SQL. However, these approaches often treat schema linking, example retrieval, and result verification as isolated stages guided primarily by surface-level semantics, leading to schema topology ambiguity, retrieval mismatch, and execution-level semantic blind spots when handling complex and heterogeneous schemas of enterprise databases. To address these issues, this paper proposes a Relational and Semantic Topology-Aware Refinement (RSTAR) framework for Text-to-SQL. First, a relational topology-aware schema linker is designed to constrain LLM reasoning to valid foreign-key navigation paths for eliminating hallucinated intermediate-table joins under schema topology ambiguity. Second, a semantic topology graph retriever is proposed to disentangle structural topologies from domain entities for resolving retrieval mismatch via structurally isomorphic few-shot retrieval across heterogeneous databases. Third, an execution-feedback refinement agent is built to convert sandbox execution signals into localized value-aware feedback for repairing execution-level semantic errors. Extensive experiments on five benchmark datasets across several mainstream LLMs demonstrate that RSTAR significantly outperforms state-of-the-art Text-to-SQL competitors, attaining 95.4% execution accuracy on Spider and 79.2% on BIRD and surpassing the strongest baseline by 6.9 points on UniSQL. The source code and datasets are submitted as Supplementary Materials for Reproducibility.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Text-to-SQL, Large Language Models, Graph Neural Networks, Retrieval-Augmented Generation, Structured Reasoning, Contrastive Learning, AI Agents, Execution Feedback

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PZnXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes RSTAR, a closed-loop Text-to-SQL framework with three modules: a topology-aware schema linker, a retriever, and an execution-feedback refinement agent. Experiments on multiple benchmarks demonstrate that RSTAR achieves strong results and surpasses prior state-of-the-art methods.

Critical strengths

S1.​ The paper identifies three concrete failure modes in enterprise Text-to-SQL (schema topology ambiguity, retrieval mismatch, execution-level blind spots) and designs dedicated modules for each.

Critical weaknesses

W1.​ The topological waveform fingerprint is presented as the key innovation of the retriever, yet the paper fails to compare it against standard retrieval baselines such as BM25 or dense retrieval (e.g., Sentence-BERT). Without these comparisons, the claimed superiority of the fingerprint-based retrieval is unsubstantiated.

W2.​ The RGAT encoder's training objective and supervision signals are underspecified. Moreover, no ablation study replaces RGAT with simpler alternatives (GCN, RGCN, vanilla GAT) to isolate its contribution. Combined with the questionable practice of using raw cross-attention scores as fingerprints without calibration, it is difficult to determine whether the reported gains stem from the graph encoder, the fingerprint design, or other components.

Questions for authors

Q1.​ Did you compare your topological fingerprint retrieval against BM25 or dense retrieval? If not, how do you rule out that a simple baseline would match or exceed your results?

Q2.​ What is the exact supervision signal for RGAT training (e.g., node classification, link prediction, contrastive learning)? How are positive and negative pairs defined, and is RGAT trained end-to-end with the retriever?

Q3.​ Raw cross-attention scores are known to be unstable and may not reflect true relevance. Did you experiment with any calibration (e.g., through contastive learning) before forming the fingerprint? An ablation study comparing raw attention fingerprints against calibrated ones would directly address this concern.

1 comment
AuthorsOpenReview ↗

Response to Reviewer PZnX

Dear Reviewer PZnX,

Thank you for the precise review. We ran all requested experiments and report the results as they are, including where they qualify our claims. Table/section numbers refer to the revised PDF.

W1/Q1: Comparison vs. BM25 / dense retrieval (retriever ladder)

Spider dev (1,034) + BIRD dev (1,534), XiYanSQL-14B, identical downstream pipeline. Recall@k hit = SQL-skeleton match with gold (skeletonization: App. A.2.2); EX also reported.

RetrieverSpider R@1/3/5BIRD R@1/3/5Spider EXBIRD EX
BM2521.9/36.9/45.128.0/39.8/47.490.870.8
BGE dense17.7/28.7/35.730.6/45.4/51.690.469.9
BGE+RGAT21.9/39.1/47.819.0/32.9/39.391.466.6
BGE+RGAT (mean-pool)21.3/38.7/47.819.6/34.0/41.190.668.3
STGL-SAR (full)25.8/41.9/50.915.4/26.1/31.691.367.1

Spider: each component contributes as claimed—RGAT + skeleton contrastive lifts R@5 35.7→47.8, the fingerprint adds +3.1; the full retriever is best on every metric. EX gaps (≤1.0) are within noise: the linker/generator compensate for retrieval noise, so Recall@k is the informative metric.

BIRD: the ordering inverts; we investigated rather than papering over it. Our fingerprint spans a fixed 10-table×20-column window, and 54.5% of BIRD databases (60.8% of questions) exceed it (Table 7)—avg 70 columns/DB, 231 columns + FK edges truncated—vs 13.9% on Spider. The topology the RGAT/fingerprint rely on is largely absent at input time on BIRD; the degradation is caused by the window, not the topology-aware principle. Implications: (1) our Table 1 BIRD result (76.66% EX) came from this same windowed retriever—it is a lower bound; (2) adaptive/hierarchical windows are now explicit future work (Sec. 5). Full ladder + coverage stats: App. A.5.2 (Table 8); protocol in A.5.1.

W2/Q2: What supervises the RGAT?

Apologies for the ambiguity—no node/link-prediction labels; supervision is entirely at the representation level, in two phases (implementation-verified):

  • Phase I (Eq. 13–17). Schema → heterogeneous graph (table–column TC, PK–FK edges). RGAT layers + gated fusion + question→schema cross-attention are trained end-to-end with a Smooth L1 loss aligning the schema-aware representation z_stage1 with the BGE embedding of the gold SQL skeleton (literals/identifiers stripped per App. A.2.2). The gold skeleton is the structural target forcing the encoder to expose join topology.
  • Phase II (Eq. 18–19). Phase-I encoder frozen. Causal Transformer + MLP head maps [question; z_stage1] to a 1024-d intent vector; (i) InfoNCE in-batch loss (temperature τ) + (ii) auxiliary hinge loss over positive pairs: same SQL skeleton, different databases. L_stage2 = L_CL + 0.3·L_sim. Sequential (not joint) optimization, which we found more stable.

GNN-variant ablation (new, single runs, identical budget):

GNNSpider R@1/3/5BIRD R@1/3/5
GCN30.4/45.6/55.418.8/31.2/37.2
RGCN25.9/41.4/49.118.6/29.7/34.4
GAT25.5/41.4/47.819.0/28.2/33.5
RGAT (ours)27.0/42.5/50.915.4/26.5/31.7

On Spider RGAT is competitive with the best variant within training variance (6K samples, batch 8); on BIRD all graph variants degrade and GCN's parameter-free smoothing is most robust—consistent with truncation, since relation-specific attention cannot manifest when half the schemas lose FK edges. Hyperparameters (LR, τ) were tuned with RGAT (Fig. 12), not re-tuned per variant. Claims softened: the revision states "topology-aware schema encoding" with RGAT as our instantiation (App. A.5.3, Table 9).

W3/Q3: Calibration — Raw vs. Calibrated

Correct: the submitted fingerprint uses raw head-averaged weights. We renormalize with softmax(logits/τ) before sort-and-concatenate and swept τ ∈ {0.25, 0.5, 1.0, 2.0, 4.0}:

Spider R@1/3/5Spider EXBIRD R@1/3/5BIRD EX
Raw (τ=1.0)25.8/41.9/50.991.315.4/26.1/31.667.1
Calibrated (τ*=0.25)29.1/45.6/55.492.614.5/26.4/32.067.0

Clear gain where the fingerprint is informative (Spider: +3.7 R@3, +1.3 EX); neutral where the window truncates (BIRD, within noise). We adopt τ*=0.25 for both datasets (App. A.5.6, Table 12). Raw attention is indeed miscalibrated; the fix is one dev-tuned scalar and improves the deployed metric.

Summary of changes: (i) new App. A.5 (Tables 7–12) under an explicit protocol; (ii) Sec. 3.3/A.2.2 rewritten with the exact Phase-I/II objectives; (iii) RGAT claims scoped; coverage statistics motivate window-adaptive fingerprints as future work. Your scrutiny led to what we consider the revision's most interesting analysis—thank you.

Reviewer_SbK8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RSTAR is a Text-to-SQL framework that aims to improve schema selection, example retrieval, and SQL repair. It targets errors caused by missing join paths, examples with unsuitable SQL structures, and mismatches between generated values and stored data. The framework has three components. A schema linker uses foreign-key relations to guide the selection of tables, columns, and join paths. It is trained through supervised fine-tuning, multi-task learning, and GRPO. A retriever learns from schema graphs and SQL skeletons. It combines learned representations with attention fingerprints to select examples with similar query structures across databases. A repair agent then executes the generated SQL and checks for errors or empty results. It examines query conditions and searches for similar values in the database to guide revisions. The evaluation covers Spider, BIRD, Spider 2.0, BEAVER, and UniSQL with five LLM backbones. Table 1 reports best execution accuracies of 95.4% on Spider and 79.2% on BIRD. The paper reports gains over the selected baselines and lower overall performance when major components are removed. Additional experiments examine retrieval quality and the time cost of refinement.

Critical strengths
  1. Useful problem setting. The paper addresses common Text-to-SQL errors, including missing joins, poor example selection, and wrong database values. Improving these steps has practical value.

  2. Broad evaluation. The experiments cover several benchmarks and LLMs. The reported gains support the usefulness of the full system across different settings.

Critical weaknesses
  1. Limited novelty. SchemaRAG already uses the same training stages, task ratio, and reward weights for schema matching. ReFSQL and SQLens also cover key ideas in retrieval and repair. The paper does not clearly explain its advances over these methods.

  2. Limited tests of the new designs. The main ablations remove whole modules. They do not show the separate gains from graph navigation, attention fingerprints, or value probing. Also, the repair step accepts SQL that runs and returns rows, which can leave wrong answers uncorrected.

Questions for authors
  1. Which parts are new compared with SchemaRAG? Can you show their gains under the same training and testing setup?

  2. How are attention fingerprints compared across schemas of different sizes? What evidence shows that their similarity reflects join structure?

  3. How does the agent fix wrong SQL that still returns rows? In Figure 1(c), Citizenship != 'French' would retain all five displayed values. How would this error trigger repair?

1 comment
AuthorsOpenReview ↗

Response to Reviewer SbK8

Dear Reviewer SbK8,

Thank you for the constructive review. We respond point by point; table/section numbers refer to the revised PDF (changes highlighted in blue).

W1/Q1: Novelty vs. SchemaRAG, ReFSQL, SQLens

We apologize for the unclear positioning. The SFT→MTL→GRPO recipe is adopted from prior work (cited) and is not claimed as our contribution; our contribution is what is trained, what is retrieved, and how the stages close into one loop:

AspectSchemaRAGReFSQL/SQLensRSTAR (ours)
Schema linkingRetrieval augmentation; no learned FK-path reasoningNot addressedLearned 4-stage navigation CoT on valid FK paths + GRPO reward on schema-element overlap (TP+2.0/FP−0.5/FN−3.0/F1+0.5)
RetrievalStructure-aware example selectionRetrieval for refinement contextRGAT schema-graph encoding + SQL-skeleton contrastive learning; topological waveform fingerprint (sorted cross-attention weights) fused with intent similarity (α=0.4)
Execution feedbackNot closed-loopMemory-augmented refinementSandbox w/ 10-s C-level timeout; AST-level value fuzzy probing (difflib); hierarchical repair (value→structure→rewrite), T_max=3
IntegrationRetrieval onlyRepair onlySingle closed loop: linking constrains generation, retrieval grounds structure, sandbox feeds back

RSTAR beats SchemaRAG under identical backbones: +2.1 EX Spider (XiYanSQL-14B: 95.4 vs 93.3), +10.8 EX BIRD (DeepSeek-v3: 79.2 vs 68.4), +6.9 UniSQL (53.57 vs 46.88); 210/0 win-loss, Wilcoxon p<0.05 (Tables 1, 4). → Revision: this table added as Table 3 (App. A.1); novelty statement sharpened in Sec. 1.

W2: Fine-grained ablations (new App. A.5; protocol in A.5.1; Spider/BIRD dev, XiYanSQL-14B)

  • Retriever ladder (Table 8, A.5.2), skeleton-hit Recall@k + EX. Spider: dense 35.7 → +RGAT 47.8 → +fingerprint 50.9 R@5; full retriever best on R@1/3/5/MRR. On BIRD the topology components degrade; the cause is quantified: 60.8% of BIRD dev questions exceed our fixed 10-table×20-column fingerprint window (Table 7; 13.9% on Spider; 231 columns + FK edges truncated). We report this as an operating-range finding: the Table 1 numbers came from this same windowed retriever, so they are a lower bound; window-adaptive fingerprinting is future work (Sec. 5).
  • GNN variants (Table 9, A.5.3): GCN/RGCN/GAT/RGAT within a few points (Spider R@5 47.8–55.4) → robust to the GNN instantiation; all degrade together on BIRD, consistent with the window analysis.
  • Path-constraint ablation (Table 10, A.5.4): disabling FK path validation/completion drops Table F1 by 8.8/8.2 and Column F1 by 14.4/18.3; the effect concentrates in Column Recall (+22.5/+30.0 from FK completion)—FK propagation mainly recovers join columns, exactly the Sec. 3.2 mechanism.
  • Value-probing ablation (Table 11, A.5.5): disabling fuzzy probing (the agent keeps the Empty/Error signal) reduces EX 95.4→94.0 (Spider), 76.7→75.8 (BIRD); the probe-free agent triggers on 5.5%/7.5% of queries at 1.1–1.2 avg turns—cost stays bounded.

Q2: Fingerprint vs. model size; structural evidence

The fingerprint is produced by the lightweight retriever (BGE + RGAT + cross-attention), not the backbone LLM—its dimensionality (10 + 10×20 slots, 210-d after sorting) and cost are independent of generator size; all Table 1 backbones share it. Structural evidence: (i) PCA (Fig. 11): embeddings cluster by SQL structure after contrastive training (Silhouette −0.035 → +0.005); (ii) the ladder isolates the fingerprint's marginal gain on in-window schemas (Spider R@5 47.8→50.9; R@1 21.9→25.8; +15.2 R@5 over dense); (iii) temperature calibration sharpens it (Spider R@3 41.9→45.6, EX 91.3→92.6; Table 12)—impossible if the sorted weights carried no structural signal. Full fingerprint↔join-overlap correlation study: camera-ready.

W3/Q3: Wrong-but-executable SQL

Honest answer: EFRA triggers only on Empty/Error, so a wrong query returning rows passes silently—a genuine limitation, now discussed in Sec. 5. Two remarks: (i) the common subclass of value errors ('French' vs 'France') surfaces as Empty and is exactly what probing repairs (Fig. 3); (ii) mitigation direction: a result-set plausibility gate (cardinality checks vs column statistics, NULL-ratio/distribution probes, aggregation cross-checks)—a soft trigger instead of a binary state. We present it as future work, not a validated mechanism.

Figure 1(c)

Our editing error—thank you. The intended example: generated literal 'French' mismatches stored 'France' → 0 rows → repaired by probing (as in Fig. 3). The submitted figure wrongly used the negated form. Corrected: the question asks for singers who are French citizens; GT filters = 'France'; the error filters = 'French' → 0 rows.

We hope these additions address your concerns and warrant a higher score.

Reviewer_MLW5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RSTAR, a closed-loop Text-to-SQL framework that unifies relational topology-aware schema linking, semantic topology-aware retrieval, and execution-feedback refinement. It effectively addresses schema topology ambiguity, retrieval mismatch, and execution-level semantic blind spots across complex enterprise databases.

Critical strengths
  1. Architectural Novelty: Unifies schema linking, retrieval, and verification under a single topological substrate within a cohesive closed-loop framework.

  2. Comprehensive Evaluation: Demonstrates state-of-the-art performance across multiple challenging benchmarks, supported by thorough ablations and analysis.

Critical weaknesses

The topology-aware design relies heavily on explicit foreign-key constraints, which may limit generalization on sparse or poorly documented databases.

Questions for authors
  1. How does the system's overall throughput scale when encountering frequent execution timeouts or expensive queries in large databases?

  2. How does the topology-aware navigation mechanism gracefully degrade or adapt when dealing with databases that lack explicit foreign-key constraints?

2 comments
AuthorsOpenReview ↗

Response to Reviewer MLW5

Dear Reviewer MLW5,

Thank you for the careful reading. We address both questions below; table/section numbers refer to the revised PDF.

Q1: Throughput scaling with large databases and expensive queries

The refinement agent is conditionally triggered and strictly bounded, not an always-on loop:

  1. Conditional triggering. EFRA activates only when the sandbox returns Empty or Error (Sec. 3.4.3, Eq. 5). A query that succeeds on the first pass incurs exactly one LLM call + one sandbox execution—zero refinement overhead. Value probing runs only on Empty states, only for the P literal predicates in WHERE/HAVING.

  2. Bounded iterations and execution time. The loop is capped at T_max = 3; every sandbox execution is guarded by a 10-s hard timeout enforced at the SQLite C layer via sqlite3.interrupt() (Appendix A.2.3), so an explosive query (e.g., an unintended cross-join) is physically terminated and returned as an EXECUTION_TIMEOUT signal. Total cost: O((1+T_max)·C_LLM + T_max·C_Exec + T_max·P·(N_max + M_max·L)) (Eq. 20)—linear, no exponential blow-up.

  3. Measured overhead. On UniSQL (Table 6), the full refinement agent adds +15.0 s/req average latency; the retriever adds only +0.04 s/req. Cost concentrates exactly on the hard tail of queries that fail initially—paid only where accuracy is recovered. The sandbox uses a multi-worker architecture with one SQLite connection per worker, so refinement turns parallelize across requests.

→ Revision: a "Scalability and Latency Budget" paragraph in Sec. 3.4 / A.2.4; per-benchmark trigger rates in the camera-ready.

Q2: Degradation when explicit foreign keys are absent

A fair limitation; we are transparent about it:

  1. Scope. All five benchmarks define explicit PK–FK constraints (representative of curated enterprise warehouses); FK-sparse databases were not stress-tested—now stated in Sec. 5.

  2. Graceful degradation, not hard failure. FK structure is an additive constraint layer, not a prerequisite: (i) the linker's 4-stage navigation CoT (anchor→path→attribute→aggregation) is learned via SFT/MTL/GRPO and still yields anchors/attributes without FK edges—only path validation degenerates to semantic plausibility; (ii) in the retriever the schema graph reduces to table–column edges and RGAT/gated fusion still operate. The system degrades toward a strong semantic-matching baseline.

  3. Mitigation. For FK-free schemas, join structure can be recovered by LLM-inferred or value-overlap edges (cf. SchemaGraphSQL), after which the same machinery applies.

New evidence on FK structure (Table 10, App. A.5.4): disabling FK path validation/completion reduces Column Recall by 22.5 (Spider) / 30.0 (BIRD) points; Table F1 −8.8/−8.2—FK propagation is the primary driver of join-column recovery.

→ Revision: (i) Sec. 5 expanded with the FK-sparse discussion; (ii) quantitative FK-removal degradation analysis committed for the camera-ready (re-running the linker over modified schemas exceeds the rebuttal window).

We hope this addresses your concerns and are grateful for the constructive feedback.

Thank you for your responses and they have addressed my concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Response: Summary of Revisions

Dear Reviewers and Area Chair,

Thank you for the constructive reviews. We summarize the revisions below; all changes are highlighted in blue in the revised PDF, and detailed point-by-point responses are posted under each review.

  1. New Appendix A.5 "Additional Fine-Grained Analyses" (evaluation protocol in A.5.1):
    • Schema-truncation statistics under the fingerprint window (Table 7) [R3-W1]
    • Retriever ladder (BM25 / BGE / BGE+RGAT / +mean-pool / full) with Recall@k and downstream EX (Table 8) [R3-W1, R2-W2]
    • GNN-encoder variants: GCN / RGCN / GAT / RGAT (Table 9) [R3-W2]
    • Linker path-constraint ablation: Table F1 −8.8/−8.2, Column Recall +22.5/+30.0 from FK completion (Table 10) [R2-W2]
    • Value-probing ablation (Table 11) [R2-W2]
    • Fingerprint calibration, Raw vs. Calibrated: Spider R@3 41.9→45.6, EX 91.3→92.6 (Table 12) [R3-W3]
  2. Figure 1(c) corrected (negation error in the question and ground-truth SQL; now consistent with the repair walkthrough in Figure 3). [R2]
  3. Sec. 3.3 / A.2.2: the retriever's supervision signals stated precisely (Phase I: Smooth-L1 alignment to gold SQL-skeleton embeddings; Phase II: frozen encoder + InfoNCE + hinge loss over skeleton-matched pairs). [R3-W2]
  4. Sec. 5: two new limitation discussions—wrong-but-executable SQL (plausibility-gate future work) and fingerprint-window coverage (60.8% of BIRD dev questions exceed the 10-table/20-column window vs. 13.9% on Spider). [R2-W3, R3-W1]
  5. Positioning table against SchemaRAG / ReFSQL / SQLens (Table 3, Appendix A.1). [R2-W1]

We remain available for any follow-up questions before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.