- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
DARES proposes a decision-aligned process supervision framework for multi-turn Text-to-SQL agents. It addresses the limitation of trajectory-level execution rewards and step-level process rewards, which may provide overly coarse credit assignment. DARES introduces Automatic Reward Unit Discovery (ARUD), which identifies semantic decisions from policy rollouts, constructs localized correct–incorrect SQL contrasts, validates them through controlled execution, and aligns validated decisions with policy-generated token spans. A decision-aligned process reward model (DA-PRM) is then trained to provide localized supervision during GRPO, while execution rewards retain global trajectory-level feedback. Experiments on SQL generation, debugging, and interactive Text-to-SQL show consistent improvements over execution-only and step-level reward baselines.
Critical strengths
- The proposed ARUD automatically discovers semantic SQL decisions and aligns them directly to policy token spans, instead of relying solely on sparse terminal execution rewards or arbitrary fixed-step boundaries. This offers a well-motivated and causally grounded credit assignment mechanism.
- The hybrid reward design—combining global execution feedback with localized decision-aligned PRM during GRPO—shows meaningful empirical gains over pure execution-based RL baselines.
Critical weaknesses
- Several external Text-to-SQL baselines in Table 1 show substantial performance drops compared to their original publications (e.g., OmniSQL-7B drops to 45.02% EX vs. 69.04% in official/native evaluations). Forcing externally trained models into the multi-turn tool-interactive harness (App. F.2–F.3) likely introduces severe protocol mismatch, conflating harness incompatibility with model capability.
- The un-finetuned Base model (Qwen3.5-9B) achieves only 38.33% EX on BIRD, significantly lower than typical zero-shot performances reported in literature for similar or even smaller models (e.g., In SQL-R1, Qwen2.5-Coder-7B reaches 58.2% in bird without reasoning and reaches 63.1% with SQL-R1 inference). Forcing the raw Base model into the strict multi-turn tool-calling format likely results in severe formatting/submission penalties rather than SQL logic failures. Consequently, the reported gains over the Base model may primarily reflect format adaptation to this specific harness rather than genuine improvements in SQL reasoning.
- The multi-turn evaluation relies heavily on SParC and CoSQL (both from 2019, built on Spider), which are largely saturated and too simple for modern LLMs. To truly demonstrate the robustness and real-world utility of the proposed agentic framework, evaluation on modern, long-horizon enterprise benchmarks like Spider 2.0 is necessary.
References
- Yao, Z. et al. Arctic-Text2SQL-R1: Simple Rewards, Strong Reasoning in Text-to-SQL, Findings of the Association for Computational Linguistics: ACL 2026, pp. 26966–26995, 2026.
- Yang, H. et al. MARS-SQL: A Multi-Agent Reinforcement Learning Framework For Text-To-SQL, Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.
- Pourreza, M. et al. Reasoning-SQL: Reinforcement Learning with SQL Tailored Partial Rewards for Reasoning-Enhanced Text-to-SQL, In Second Conference on Language Modeling (COLM), 2025.
- Ma, P. et al. SQL-R1: Training Natural Language to SQL Reasoning Model By Reinforcement Learning, In Advances in Neural Information Processing Systems (NeurIPS), 2025.
- Lei, F. et al. Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows, In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Questions for authors
- Could the authors report these specialized baselines under their native/author-recommended inference protocols alongside the DARES harness to decouple method gains from protocol mismatch?
- What are the exact data source, schema distribution, and construction pipeline for the 22,477 RL prompts? Are there any potential train-test database overlaps with the evaluation benchmarks?
- Why is the Base model's performance on standard SQL generation so low? If the Base model is prompted with standard zero-shot direct generation or native reasoning (e.g., standard CoT or SQL-R1 style inference), does it outperform the multi-turn harness numbers reported in Table 2?
- SParC and CoSQL are largely saturated. Have the authors evaluated DARES on more challenging, modern multi-turn/agentic benchmarks such as Spider 2.0-lite? If so, how does the decision-aligned PRM perform in such long-horizon environments?
2 comments
Thank you for the response and additional experiments. However, my core concerns regarding evaluation validity remain unresolved:
- Implausible External Baseline Results: The newly reported "native protocol" scores (e.g., 42.93% for OmniSQL-7B and 45.41% for SQL-R1-14B on BIRD) still contradict well-established community benchmarks (typically 60%+), and the Base model's zero-shot/CoT scores (24%–35%) remain anomalously low. This strongly indicates systemic flaws in the authors' prompt formatting, schema representation, or evaluation pipeline, invalidating comparisons against external SOTA models.
- Limited Generalization: On the challenging Spider 2.0-lite subset, DA-PRM yields no improvement over Step PRM (both 9.63%).
While internal ablations show relative gains, the flawed external comparisons prevent me from verifying the true effectiveness of the method. Therefore, I will maintain my score of 2.
Thank you for the insightful questions. Below are the additional evaluations and data audit.
Q1:
We added native generation protocol alongside Table 1's shared harness, evaluating each model on 1,534 BIRD, 1,203 SParC and 1,063 CoSQL examples with a local SQLite scorer. BIRD EX:
On BIRD, four of the five baselines achieve comparable or higher EX under the shared harness, with an average difference of only 0.32 percentage points across all models, indicating no systematic disadvantage introduced by the shared protocol. However, XiYanSQL-14B improves by 3.69 points under its native protocol, highlighting model-specific protocol sensitivity. Reporting both settings therefore helps distinguish protocol-dependent variations from genuine methodological improvements and ensures a more transparent comparison.
Q2:
The 22,477 prompts come from official train-lineage episodes:
SParC and CoSQL share 140 schemas, giving 209 unique database IDs overall. Prompts combine schema, current request and prediction-derived history/state; gold SQL is used on the reward side. Construction uses a deterministic 5% episode holdout and filters excessive length, visible gold SQL and unreliable execution references.
The audit identifies cross-version
formula_1database reuse: 131 SParC/CoSQL RL prompts, 174 BIRD dev questions and 44 BIRD-CRITIC issues use related versions with shared tables and overlapping rows. Normalized question strings match 0/1,534 BIRD, 90/1,203 SParC and 71/1,063 CoSQL evaluation rows, ignoring database/dialogue context. There is no train-test database overlaps with the evaluation benchmarks.Q3:
We tested three generation protocols on 3,800 BIRD/SParC/CoSQL public-dev examples:
Compared with zero-shot direct generation, both CoT and SQL-R1-style reasoning substantially improve performance, suggesting that the Base model benefits from explicit reasoning during SQL generation. However, neither reasoning protocol outperforms the multi-turn harness reported in Table 2. Specifically, the multi-turn harness exceeds the strongest alternative (CoT) by 3.31, 4.67, and 4.71 percentage points on BIRD, SParC, and CoSQL, respectively. These results indicate that the relatively low Base performance is not caused by an unfavorable multi-turn evaluation protocol; rather, the multi-turn harness provides a stronger inference setting than standard direct generation or standalone reasoning.
Q4:
As Snowflake service for Spider 2.0-lite is not recovered, we evaluated DARES on 135 locally executable SQLite tasks from Spider 2.0-lite (135/547), a more challenging benchmark involving complex, agentic SQL workflows. DA-PRM achieves 13/135 execution matches (9.63%), outperforming Base CoT and Exec-RM (both 7/135, 5.19%) as well as SFT (11/135, 8.15%), while matching Step PRM (13/135, 9.63%). These results suggest that process-level reward modeling provides benefits over outcome-only supervision in more challenging environments. However, DA-PRM does not show additional gains over Step PRM in this setting, highlighting the remaining challenges of decision-aligned credit assignment on complex, long-horizon tasks. Since this evaluation covers only the locally executable SQLite subset under direct-adapter inference, we regard these findings as preliminary rather than conclusive evidence of long-horizon agentic generalization. We will evaluate DARES and report the results on Spider 2.0-lite immediately after the Snowflake service recovers.