acceptodds
Under review as a conference paper at ICLR 2027

DARES: Decision-Aligned Process Supervision for Multi-Turn Text-to-SQL Agents

Abstract

Multi-turn Text-to-SQL agents must translate evolving user intents and database feedback into a sequence of tool-use and query decisions. Terminal execution rewards indicate whether a trajectory succeeds but not which decisions were responsible. Step-level rewards offer finer credit, yet their predefined step boundaries need not align with the semantic decisions that determine SQL correctness: one step may mix correct and faulty decisions, and one decision may span several steps. We propose DARES, which automatically discovers decision-aligned reward units from policy rollouts. Its Automatic Reward Unit Discovery (ARUD) constructs localized SQL contrasts, each pairing a correct and a faulty realization of one decision, verifies through controlled execution that the correct side indeed yields better outcomes, and maps the verified decisions back to the policy-generated token spans that express them. A decision-aligned process reward model learns from these contrasts and provides localized credit during GRPO, while execution rewards retain trajectory-level supervision. On interactive Text-to-SQL, DARES improves normalized reward over execution-only GRPO by up to points. The result show that decision-aligned process supervision improves policy optimization across diverse multi-turn database tasks.

Primary area:
reinforcement learning
Keywords:
Text-to-SQL, Agentic Reinforcement Learning, Process Reward Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5pGTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

DARES proposes a decision-aligned process supervision framework for multi-turn Text-to-SQL agents. It addresses the limitation of trajectory-level execution rewards and step-level process rewards, which may provide overly coarse credit assignment. DARES introduces Automatic Reward Unit Discovery (ARUD), which identifies semantic decisions from policy rollouts, constructs localized correct–incorrect SQL contrasts, validates them through controlled execution, and aligns validated decisions with policy-generated token spans. A decision-aligned process reward model (DA-PRM) is then trained to provide localized supervision during GRPO, while execution rewards retain global trajectory-level feedback. Experiments on SQL generation, debugging, and interactive Text-to-SQL show consistent improvements over execution-only and step-level reward baselines.

Critical strengths
  1. The proposed ARUD automatically discovers semantic SQL decisions and aligns them directly to policy token spans, instead of relying solely on sparse terminal execution rewards or arbitrary fixed-step boundaries. This offers a well-motivated and causally grounded credit assignment mechanism.
  2. The hybrid reward design—combining global execution feedback with localized decision-aligned PRM during GRPO—shows meaningful empirical gains over pure execution-based RL baselines.
Critical weaknesses
  1. Several external Text-to-SQL baselines in Table 1 show substantial performance drops compared to their original publications (e.g., OmniSQL-7B drops to 45.02% EX vs. 69.04% in official/native evaluations). Forcing externally trained models into the multi-turn tool-interactive harness (App. F.2–F.3) likely introduces severe protocol mismatch, conflating harness incompatibility with model capability.
  2. The un-finetuned Base model (Qwen3.5-9B) achieves only 38.33% EX on BIRD, significantly lower than typical zero-shot performances reported in literature for similar or even smaller models (e.g., In SQL-R1, Qwen2.5-Coder-7B reaches 58.2% in bird without reasoning and reaches 63.1% with SQL-R1 inference). Forcing the raw Base model into the strict multi-turn tool-calling format likely results in severe formatting/submission penalties rather than SQL logic failures. Consequently, the reported gains over the Base model may primarily reflect format adaptation to this specific harness rather than genuine improvements in SQL reasoning.
  3. The multi-turn evaluation relies heavily on SParC and CoSQL (both from 2019, built on Spider), which are largely saturated and too simple for modern LLMs. To truly demonstrate the robustness and real-world utility of the proposed agentic framework, evaluation on modern, long-horizon enterprise benchmarks like Spider 2.0 is necessary.

References

  • Yao, Z. et al. Arctic-Text2SQL-R1: Simple Rewards, Strong Reasoning in Text-to-SQL, Findings of the Association for Computational Linguistics: ACL 2026, pp. 26966–26995, 2026.
  • Yang, H. et al. MARS-SQL: A Multi-Agent Reinforcement Learning Framework For Text-To-SQL, Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.
  • Pourreza, M. et al. Reasoning-SQL: Reinforcement Learning with SQL Tailored Partial Rewards for Reasoning-Enhanced Text-to-SQL, In Second Conference on Language Modeling (COLM), 2025.
  • Ma, P. et al. SQL-R1: Training Natural Language to SQL Reasoning Model By Reinforcement Learning, In Advances in Neural Information Processing Systems (NeurIPS), 2025.
  • Lei, F. et al. Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows, In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Questions for authors
  1. Could the authors report these specialized baselines under their native/author-recommended inference protocols alongside the DARES harness to decouple method gains from protocol mismatch?
  2. What are the exact data source, schema distribution, and construction pipeline for the 22,477 RL prompts? Are there any potential train-test database overlaps with the evaluation benchmarks?
  3. Why is the Base model's performance on standard SQL generation so low? If the Base model is prompted with standard zero-shot direct generation or native reasoning (e.g., standard CoT or SQL-R1 style inference), does it outperform the multi-turn harness numbers reported in Table 2?
  4. SParC and CoSQL are largely saturated. Have the authors evaluated DARES on more challenging, modern multi-turn/agentic benchmarks such as Spider 2.0-lite? If so, how does the decision-aligned PRM perform in such long-horizon environments?
2 comments

Thank you for the insightful questions. Below are the additional evaluations and data audit.

Q1:

We added native generation protocol alongside Table 1's shared harness, evaluating each model on 1,534 BIRD, 1,203 SParC and 1,063 CoSQL examples with a local SQLite scorer. BIRD EX:

Modelnative protocolShared harness
OmniSQL-7B42.93%45.02%
SQL-R1-7B43.59%44.32%
SQL-R1-14B45.41%47.72%
XiYanSQL-14B48.74%45.05%
MTSQL-R1-4B32.61%32.79%

On BIRD, four of the five baselines achieve comparable or higher EX under the shared harness, with an average difference of only 0.32 percentage points across all models, indicating no systematic disadvantage introduced by the shared protocol. However, XiYanSQL-14B improves by 3.69 points under its native protocol, highlighting model-specific protocol sensitivity. Reporting both settings therefore helps distinguish protocol-dependent variations from genuine methodological improvements and ensures a more transparent comparison.

Q2:

The 22,477 prompts come from official train-lineage episodes:

SourcePromptsDatabasesTables per database (median)Columns per database (median)
BIRD6,73969534
SParC8,422140419
CoSQL7,316140419

SParC and CoSQL share 140 schemas, giving 209 unique database IDs overall. Prompts combine schema, current request and prediction-derived history/state; gold SQL is used on the reward side. Construction uses a deterministic 5% episode holdout and filters excessive length, visible gold SQL and unreliable execution references.

The audit identifies cross-version formula_1 database reuse: 131 SParC/CoSQL RL prompts, 174 BIRD dev questions and 44 BIRD-CRITIC issues use related versions with shared tables and overlapping rows. Normalized question strings match 0/1,534 BIRD, 90/1,203 SParC and 71/1,063 CoSQL evaluation rows, ignoring database/dialogue context. There is no train-test database overlaps with the evaluation benchmarks.

Q3:

We tested three generation protocols on 3,800 BIRD/SParC/CoSQL public-dev examples:

ProtocolBIRD EXSParC QEXCoSQL QEX
Zero-shot direct24.30%27.37%27.97%
CoT35.02%48.45%48.16%
SQL-R1-style33.72%48.04%46.75%
Multi-turn harness (Table 2)38.33%53.12%52.87%

Compared with zero-shot direct generation, both CoT and SQL-R1-style reasoning substantially improve performance, suggesting that the Base model benefits from explicit reasoning during SQL generation. However, neither reasoning protocol outperforms the multi-turn harness reported in Table 2. Specifically, the multi-turn harness exceeds the strongest alternative (CoT) by 3.31, 4.67, and 4.71 percentage points on BIRD, SParC, and CoSQL, respectively. These results indicate that the relatively low Base performance is not caused by an unfavorable multi-turn evaluation protocol; rather, the multi-turn harness provides a stronger inference setting than standard direct generation or standalone reasoning.

Q4:

As Snowflake service for Spider 2.0-lite is not recovered, we evaluated DARES on 135 locally executable SQLite tasks from Spider 2.0-lite (135/547), a more challenging benchmark involving complex, agentic SQL workflows. DA-PRM achieves 13/135 execution matches (9.63%), outperforming Base CoT and Exec-RM (both 7/135, 5.19%) as well as SFT (11/135, 8.15%), while matching Step PRM (13/135, 9.63%). These results suggest that process-level reward modeling provides benefits over outcome-only supervision in more challenging environments. However, DA-PRM does not show additional gains over Step PRM in this setting, highlighting the remaining challenges of decision-aligned credit assignment on complex, long-horizon tasks. Since this evaluation covers only the locally executable SQLite subset under direct-adapter inference, we regard these findings as preliminary rather than conclusive evidence of long-horizon agentic generalization. We will evaluate DARES and report the results on Spider 2.0-lite immediately after the Snowflake service recovers.

Thank you for the response and additional experiments. However, my core concerns regarding evaluation validity remain unresolved:

  1. Implausible External Baseline Results: The newly reported "native protocol" scores (e.g., 42.93% for OmniSQL-7B and 45.41% for SQL-R1-14B on BIRD) still contradict well-established community benchmarks (typically 60%+), and the Base model's zero-shot/CoT scores (24%–35%) remain anomalously low. This strongly indicates systemic flaws in the authors' prompt formatting, schema representation, or evaluation pipeline, invalidating comparisons against external SOTA models.
  2. Limited Generalization: On the challenging Spider 2.0-lite subset, DA-PRM yields no improvement over Step PRM (both 9.63%).

While internal ablations show relative gains, the flawed external comparisons prevent me from verifying the true effectiveness of the method. Therefore, I will maintain my score of 2.

Reviewer_9cZCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper propose DARES, which automatically discovers decision-aligned reward units from policy rollouts in text2sql task

Critical strengths

The work involves substantial implementation and experimental effort. Aligning process feedback with semantic decisions is intuitively well motivated, and the reported results support the overall effectiveness of the approach.

Critical weaknesses

W1. The paper’s novelty is limited, and its distinction from existing methods, particularly HPL [1] and GEAR [2], is not sufficiently established. The authors should clearly discuss these relationships in the related work and provide experimental comparisons to demonstrate the advantages of the proposed approach.

W2. The paper does not directly validate whether preferences derived from terminal SQL contrasts can be reliably mapped to rewards for earlier tool calls and reasoning spans. The authors should provide evidence supporting the reliability of this mapping.

[1] Gao et al. Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents. ICLR 2026.

[2] Li et al. GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation. 2026.

Questions for authors

see weakness

1 comment

Thank you for the insightful questions. We clarify the relationship to HPL/GEAR and provide direct intervention evidence for non-terminal credit.

W1:

The distinction is the unit and source of credit. HPL learns preferences over trajectories, steps and groups of consecutive actions. GEAR uses student-teacher disagreement to reweight a shared trajectory advantage over token segments. DARES aligns semantic SQL decisions to policy spans and learns decision-specific signed rewards. Within one SQL action, a correct join and an incorrect predicate can therefore receive different credit when aligned to their respective spans. We will make these relationships explicit in the related work.

We trained GEAR-style and HPL-style adaptations, and the comparison among the three baselines are below:

MethodBIRD EXSParC QEXCoSQL QEX
GEAR-style43.03%55.94%56.54%
HPL-style43.61%55.61%55.22%
DA-PRM (ours)54.88%66.75%63.19%

These results suggest the potential benefits of decision-aligned credit assignment.

We also did a case study to demonstrate the credit assignment difference among the three method: In a real SParC training trace, the agent generates:

SELECT AVG(height), AVG(weight) FROM people WHERE sex = 'Male';

The database encodes males as 'M', so the query returns (NULL, NULL), while correcting only the predicate yields the gold result. HPL compares actions or action groups without explicitly assigning decision-specific signed rewards. GEAR reweights a shared trajectory advantage using positive token weights; since all four rollouts receive zero outcome reward, its token advantages remain zero. In contrast, DARES assigns PRM-derived process advantages of +0.201 to AVG(height), +0.833 to AVG(weight), and −0.349 to sex='Male', reinforcing correct aggregates while penalizing the incorrect predicate despite identical outcome rewards.

W2:

We directly intervened on 125 retained non-terminal units and 200 randomly sampled injectable rejected candidates. For each unit, we fixed the visible prefix, injected preferred/dispreferred realizations and regenerated the remaining interaction, including tool calls and final SQL. Both groups used the same SFT policy and runtime, with eight paired continuations per arm: 5,200 generations in total.

Filter groupPositive / tie / negative effectMean Δ success
Retained (125)122 / 2 / 10.882
Rejected (200)42 / 137 / 210.086

The preferred realization improves continuation success in 122/125 retained cases (97.6%). The rejected positive fraction is 21.0% (source-cluster bootstrap 95% CI 13.3%-29.1%), conditional on rejection and injectability. An independently recomputed original-database SQL proxy yields 135 positive / 8 nonpositive intervention effects for proxy-positive cases and 29 / 153 for proxy-nonpositive cases. Mean absolute proxy errors for preferred/dispreferred arms are 0.018/0.100 in retained units and 0.198/0.224 in rejected units.

These empirical eight-rollout signs support directional consistency on the sampled non-terminal units. The retained sample contains 121 reasoning spans, primarily ORDER/LIMIT, and four tool arguments; the rejected sample contains 181 reasoning/state and 19 tool-argument units. We will include the factor, surface and turn-distance breakdowns.

Reviewer_akgWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DARES, a decision-aligned process supervision framework for multi-turn Text-to-SQL Agents, aimed at improving local credit assignment in reinforcement learning. Its ARUD discovers decision-level reward units through error injection/repair and execution verification, and trains DA-PRM to support finer-grained process supervision during GRPO. Experiments on multiple benchmarks show that this method outperforms the execution reward GRPO and Step PRM.

Critical strengths
  1. The paper addresses an important problem in local credit assignment. Compared with fixed step-level supervision, the decision-level reward unit can more accurately align semantic decisions. ARUD combines local SQL comparison and execution verification to automatically construct supervision signals.

  2. The experiments covered SQL generation, debugging, and interactive tasks. With the shared Qwen3.5-9B backbone, the same preference data, and the same training budget, DARES outperformed Step PRM on all reported controlled indicators and provided ablation and error localization analyses.

Critical weaknesses
  1. Insufficient validation of causal attribution. The SQL execution comparison in Equation (11) cannot directly represent the actual multi-turn interaction intervention effect in Equation (10). Although Appendix B provides a conditional theoretical analysis, there is a lack of direct intervention experiments and proxy error verification.

  2. Limited evidence for independent gains. Existing work (such as EXPO-SQL) has explored SQL local rewards, but this paper lacks sufficient direct comparisons. Gains over Step PRM are modest on several benchmarks, and without uncertainty estimates, their stability remains unclear.

Questions for authors
  1. Can the coverage rates of reward units for different decision types be reported, and can the consistency between the SQL agent's judgment and the actual interaction intervention results be verified?

  2. Can a local execution reward baseline be added to distinguish the contributions of semantic alignment and DA-PRM respectively?

  3. Can the authors clarify how the verifier's diagnostic head in Appendix E.2 is trained?

2 comments

Thank you for the insightful questions. We added coverage accounting, direct interventions and local-execution baselines.

Q1:

Coverage. Candidate retention by decision type is:

FactorCandidate retention
JOIN85.44%
FILTER55.53%
AGG5.32%
GROUP BY7.24%
ORDER/LIMIT83.33%
PROJECTION28.78%

Across 81,920 RL rollouts, eligible unit credit reaches 22,226/22,515 (98.72%) executable-but-wrong cases and 23.50% of their tokens (7.10% of all rollout tokens).

Intervention consistency. We fixed each visible prefix, injected preferred/dispreferred realizations and regenerated the remaining interaction, including tools and final SQL. We evaluated 125 retained units and 200 randomly sampled injectable rejected units with the same SFT policy and eight paired continuations per arm: 5,200 generations in total.

Filter groupPositive / tie / negative intervention effectMean Δ success
Retained (125)122 / 2 / 10.882
Rejected (200)42 / 137 / 210.086

Retained units have positive effects in 97.6% of sampled cases. The independently recomputed SQL-proxy/intervention matrix is 135 positive / 8 nonpositive effects for proxy-positive cases and 29 / 153 for proxy-nonpositive cases. The rejected positive fraction is 21.0% (source-cluster bootstrap 95% CI 13.3%-29.1%), conditional on rejection and injectability. These are empirical eight-rollout signs; the retained sample contains 121 reasoning spans, primarily ORDER/LIMIT, and four tool arguments.

Q2:

We trained two local-execution baselines for 512 updates: one assigns clause-execution scores to aligned decision spans; the other distributes the same scores over the entire assistant action. The table compares reward sources at each assignment granularity. All values are BIRD-CRITIC SQLite-500 success rates.

Where reward is assignedDARES StyleEXPO-SQL Style
Entire assistant action35.40% (Step PRM)34.60%
Aligned decision span37.00% (DA-PRM)32.40%

With action-level assignment, learned PRM rewards achieve 35.40% success, slightly outperforming execution-based rewards (34.60%, +0.80 pp). Notably, with decision-span alignment, DA-PRM reaches 37.00%, substantially outperforming EXPO-SQL-style rewards (32.40%, +4.60 pp). Moreover, span alignment improves learned PRM performance by 1.60 pp but reduces execution-based performance by 2.20 pp. These findings suggest that fine-grained credit assignment is most effective when paired with semantically informed, learned reward signals, whereas directly assigning clause-execution scores to individual spans may introduce noisy or misleading supervision. Thus, DARES's gains arise not merely from finer reward granularity, but from combining decision alignment with learned process rewards.

For uncertainty, 20,000-resample paired bootstrap and exact McNemar tests give:

DA-PRM minusDifference95% CI (pp)p
SFT+2.20 pp[−0.80, +5.20]0.193
Exec-RM+2.00 pp[−1.00, +5.00]0.237
Step PRM+1.60 pp[−1.40, +4.80]0.374

Two inference seeds per fixed checkpoint at temperature 0.8, top-p 0.95 and 16K give Step PRM 30.90% ± 0.42 pp, DA-PRM 31.00% ± 1.41 pp and Exec-RM 30.20% ± 1.13 pp (mean ± sample SD).

Q3:

In the current implementation, DA-PRM uses a text Qwen3.5-9B backbone with rank-8 LoRA and a scalar reward head. Training combines local pairwise softplus ranking with same-factor cross-context ranking. The diagnostic readout maps unit risk scores to aligned events, spans and semantic factors. Structured diagnosis generation uses a separate SFT policy through prompting. We will clarify the term "diagnostic head," the component-to-checkpoint correspondence and the respective training/inference roles in Appendix E.2.

Reviewer_9cZCOpenReview ↗

Thanks for replying

Thanks for the author's replying, some of my concerns are addressed. However, due to the size of additional results is limited. I am hesitate to raise my score. I will make the decision based on the replying of other reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.