- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies schema grounding and error correction in Text-to-SQL. The authors propose RSI-SQL, an inference-time multi-agent framework designed to iteratively refine database queries under database execution feedback. Operating with frozen model parameters, the framework coordinates three LLM-prompted roles: a Graph Grounding Agent that maintains hypothesis particles for schema linking, an Actor Agent that proposes candidate actions (GENERATE, REPAIR, POLL), and a Critic Agent that scores interactions to select candidate trajectories. An Action Graph tracks dependencies across a shallow search horizon (). The paper introduces continuous POMDP and Lipschitz perturbation bounds as theoretical framing, and evaluates the method primarily on BIRD-Dev, Spider-Dev, Spider-DK, and Spider2.0-SQLite using Qwen2.5-14B backbones.
Critical strengths
Sensible Operational Decomposition for Interactive Text-to-SQL: Decomposing complex database reasoning into specialized steps—schema grounding, speculative database polling, and execution-guided repair—is an intuitive operational workflow for interactive SQL generation. The qualitative breakdown of SQL error types (schema mismatch, intent ambiguity, execution semantics) provides descriptive context for understanding where iterative queries fail.
Critical weaknesses
-
Severe Empirical Inconsistency, Cherry-Picking, and Test Regressions: The paper's primary empirical claim is contradicted within the manuscript. In Table 1, the authors report a headline accuracy of 64.41% on BIRD-Dev for Qwen2.5-Coder-14B. However, in Table 2, the exact same system across five runs crashes to 58.5 ± 0.6%—a 5.91-point drop that lies nearly 10 standard deviations away from the mean. The authors explicitly concede that Table 1 represents a single run ("Each entry is one run"), indicating that the headline claim is a cherry-picked outlier; the reproducible 5-run mean (58.5%) barely beats a simple single decoder (56.9%) by 1.6 points. Furthermore, the authors quietly admit in Section 5.5 and Appendix C.2 that on the official Spider-Test set, the coordinated system actually degraded performance by 5.9 percentage points (82.1% vs. 88.0% for the single decoder), demonstrating severe instability on unseen distributions.
-
Misleading Nomenclature ("Recursive Self-Improvement") and Prohibitive Compute Cost: Branding this framework as "Recursive Self-Improvement (RSI)" represents substantial concept inflation. The backbone model is 100% frozen with zero model training, parameter updates, or policy learning (Appendix A.4); it is strictly an inference-time prompt loop (ReAct / Reflexion) unrolled for 3 steps. Despite this modest mechanism, answering a single query burns 78.4 model calls, 74.1K tokens, and 136.8 seconds (Table 3), yet the authors provide no compute-matched baseline comparisons (e.g., Best-of-78 sampling or majority voting). Finally, the continuous POMDP and Lipschitz formulations (Propositions 1 & 2) are cosmetic, as the authors concede in Appendix B.2 that they cannot numerically validate the theoretical bounds.
Questions for authors
-
Resolution of the 9.85$\sigma$ Discrepancy Between Table 1 and Table 2 and Test Regressions: How do the authors reconcile the 5.91 percentage-point collapse from 64.41% in Table 1 to 58.5 ± 0.6% in Table 2 for Qwen2.5-Coder-14B on BIRD-Dev, where 64.41% lies 9.85 standard deviations away from the 5-run mean? If Table 1 is indeed an unrepresentative single outlier, how can the paper's headline claims be justified when the reproducible mean (58.5%) barely improves over the single decoder (56.9% by +1.6 points)? Furthermore, why did the coordinated system severely regress on Spider-Test (-5.9 points below the single decoder in Appendix C.2)?
-
Compute-Matched Baseline Comparison: In the reported setting at search depth 3 (, Table 3), RSI-SQL requires 78.4 model calls and 136.8 seconds per query. How does a standard single-decoder model perform when granted an equivalent inference-compute budget (e.g., Best-of-78 sampling with execution filtering or majority voting over 78 sampled paths)?