acceptodds
Under review as a conference paper at ICLR 2027

Human-Level Text-to-SQL via RL with Verified Data and Task-Specific Rewards

Abstract

Despite increasingly complex multi-stage inference pipelines, existing text-to-SQL systems are still limited by the SQL reasoning capability of their underlying large language model (LLM). These systems still underperform human experts by over 10% on benchmarks. In this paper, we show that reinforcement learning with verified training data and task-specific rewards enables a single fine-tuned LLM to achieve human-level performance. We introduce BIRD-Platinum, a dataset of 2.5k instances sampled from BIRD Train and curated through multiple rounds of expert verification. We corrected annotation errors in 61% of the instances. We then diagnose two limitations of standard execution-based outcome rewards. First, they can reward semantically incorrect queries that coincidentally return the correct result. Second, they provide no direct incentive to use provided external knowledge. To address these limitations, we propose ReViSQL, which combines execution-based grading with bounded SQL equivalence verification and process rewards that encourage external-knowledge use. We show that fine-tuning Kimi-K2.6 on BIRD-Platinum with REVISQL matches the 92.96% human performance on Arcwise-Plat, the expert-verified variants of BIRD. Our model outperforms open-source pipelines by 9–22% and surpasses GPT 5.6 Sol Ultra and Claude Fable 5 by 3–8% at 12–15% of their inference cost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
RL, text-to-SQL, data quality

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QuF6OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces BIRD-Platinum, a roughly 2,500-example subset of BIRD Train corrected through independent expert review, and REVISQL, an RL reward that adds bounded SQL-equivalence checks and external-knowledge process penalties to execution grading. A same-backbone Qwen comparison reports gains from training on BIRD-Platinum across Arcwise-Plat and Spider2. The full Kimi-K2.6 system reaches 93.8% with greedy decoding on Arcwise-Plat-Full and 93.0% with 16-sample self-consistency on Arcwise-Plat-SQL. The paper compares these scores with a previously reported 92.96% human proxy on BIRD.

Critical strengths
  1. The data effort is substantive: independent experts corrected gold SQL in 52.1% of examples, found some annotation error in 61.1%, and rejected 16.2% of first-round proposed corrections on verification. The same-backbone training comparison also shows gains beyond BIRD on Spider2, making this more than a benchmark-specific cleanup exercise.
  2. The paper diagnoses a real weakness in execution-based RL rewards. Bounded verification refutes equivalence for an average of 32.8% of execution-accepted rollouts in a pilot run, and the complete REVISQL intervention adds 2.8–3.8 points over standard rewards when both train on BIRD-Platinum. This is useful evidence for improving the training signal, even though the individual reward components are not separately identified.
Critical weaknesses
  1. The gains are better established for the combined interventions than for their proposed causes. BIRD-Platinum is a sampled subset of BIRD Train, but the data comparison does not report a control trained on the original annotations of those same examples with the same rollout and update budget. The large gain may therefore mix annotation correction with example selection or training-set size. Likewise, the REVISQL ablation adds bounded verification, a structured prompt, and string-checked process penalties together. The small manual audit of external-knowledge failures is encouraging, but it does not separate those effects or measure semantic compliance across rewarded SQL outputs. The paper supports the value of the packages more clearly than the causal account of each ingredient.
  2. “Human-level” is a comparison across different evaluation conditions. The 92.96% human proxy was reported for BIRD, whereas the model is scored on Arcwise-Plat variants with corrected gold SQL and, in the Full variant, corrected questions and external knowledge. Those repairs improve the evaluation, but they also make the historical number an unsuitable parity threshold without a matched human assessment. The model's strong scores stand on their own; they do not yet establish the title's human-level claim.
Questions for authors
  1. From the existing runs, were the BIRD Train and BIRD-Platinum comparisons matched on example IDs and RL rollout/update budget? For REVISQL, what fraction of VeriEQL-refuted or inconclusive rollouts still received positive training advantage, and what evidence shows that high-reward SQL actually satisfies the supplied external knowledge? These results would clarify whether the reported gains support the proposed mechanisms rather than only the combined setup.
  2. The 92.96% human proxy was measured on BIRD, whereas the model is evaluated on corrected variants of that benchmark. What were the proxy's exact instances, input information, and scoring rule? Since the corrected variants change the gold SQL, and the Full variant also changes the questions and external knowledge, what evidence establishes the historical proxy as a valid human comparator? Numerical proximity between aggregate scores alone does not establish parity.
4 comments

We thank the reviewer for their comments. We addressed them below and revised the paper with changes highlighted in blue.

W1.1 / Q1.1: From the existing runs, were the BIRD Train and BIRD-Platinum comparisons matched on example IDs and RL rollout/update budget?

We clarify that the existing BIRD Train and BIRD-Platinum comparison uses the same RL rollout budget and maximum update budget. However, they do not match on example IDs. Example IDs of BIRD-Platinum is a random sample of that of BIRD-Train.

We additionally trained Kimi K2.6 on BIRD-Train-Match, which matches BIRD-Platinum on example IDs with the original annotation from BIRD Train. We kept other settings consistent. BIRD-Train-Match achieved 78.31% accuracy on Arcwise-Plat-Full and 72.89% on Arcwise-Plat-SQL, underperforming the full BIRD Train by 1.4-1.6% and BIRD-Platinum by 15.5-18.5%. Therefore, the gain of BIRD-Platinum comes from correcting the annotations, not from example selection or training-set size.

We added these results to Section 5.3 of the revised paper.

W1.2: Likewise, the REVISQL ablation adds bounded verification, a structured prompt, and string-checked process penalties together.

We ran an additional ablation study to isolate the effect of the two reward components. We find that neither component alone accounts for the gain of ReViSQL-K2.6. In the revised paper, we added the new results in Section 5.3. We summarize the key findings as follows.

Structured promptVeriEQLProcessArcwise-Plat-FullArcwise-Plat-SQL
No89.9688.55
YesYes90.56 (+0.60)88.76 (+0.21)
YesYes91.16 (+1.20)90.16 (+1.61)
YesYesYes93.78 (+3.82)91.37 (+2.82)

Compared to the standard execution-based outcome reward, process penalties alone only slightly improve accuracy (+ 0.2-0.6%), while verification-aware outcome reward alone improves it by 1.2–1.6%. Neither of them individually recovers the gain of ReViSQL-K2.6 over the standard reward.

We find that the two components address complementary failures. Even though the model trained with only verification-aware reward uses the same prompt as ReViSQL, it violates the external-knowledge verification requirement 3.8x as often as ReViSQL. Through our manual review, we also find that the share of its incorrect queries that misuse the external knowledge is 3.6x that of ReViSQL. Therefore, both components are necessary.

W1.3 / Q1.2: The small manual audit of external-knowledge failures is encouraging, but it does not separate those effects or measure semantic compliance across rewarded SQL outputs. For REVISQL, what fraction of VeriEQL-refuted or inconclusive rollouts still received positive training advantage, and what evidence shows that high-reward SQL actually satisfies the supplied external knowledge?

We analyzed the generated SQL queries and their rewards over the training of ReViSQL. We find that the fraction of VeriEQL-refuted rollouts that still receive a positive advantage drops from 39% to 5% over training, while 95–99% of high-reward queries satisfy the external knowledge. In the revised paper, we added these analyses to Section 4.2. We summarize the key findings as follows.

VeriEQL-refuted and inconclusive rollouts. We find that the fraction of VeriEQL-refuted rollouts that still receive a positive advantage decreases from 39% in the first epoch to 5% in the eighth epoch. This trend indicates that the model generates more semantically correct queries. Furthermore, ReViSQL does not penalize inconclusive verdicts, because they reflect VeriEQL's internal incompleteness. Nevertheless, the fraction of inconclusive rollouts of all rollouts stays stable throughout training, fluctuating between 5.5% and 10.1%.

External-knowledge compliance of high-reward queries. In each of the eight training epochs, we randomly sampled 100 queries that received full reward and labeled them with an LLM judge (Appendix I.1). In every epoch, we find that 95–99% of the sampled queries satisfy the external knowledge. Therefore, our reward rarely gives full credit to SQL that violates the external knowledge.

Q2: The 92.96% human proxy was measured on BIRD, whereas the model is evaluated on corrected variants of that benchmark.

We acknowledge that the historical proxy is not a valid human comparator on Arcwise-Plat benchmarks. The BIRD Team measured the 92.96% execution accuracy on the BIRD Test split, which contains different data instances from Arcwise-Plat. Therefore, we removed the human-level claim throughout the revised paper.

Nevertheless, our results demonstrate substantial accuracy gains and lower inference costs. ReViSQL-K2.6 outperforms OpenSearch, the strongest prior open-source pipeline, by 8.4% on Arcwise-Plat-SQL and 5.6% on Arcwise-Plat-Full, while reducing per-query inference cost by 37%. Compared with GPT-5.6 Sol Ultra and Claude Fable 5, it achieves 3.2–7.9% higher accuracy on Arcwise-Plat-SQL and 1.0–3.8% higher accuracy on Arcwise-Plat-Full, using only 12–15% of their inference cost.

Thank you for the additional experiments and clarifications. I have no further questions at this point.

Thank you for your thoughtful feedback and for considering our additional experiments and clarifications. We are glad to have addressed your questions. We would also be happy to clarify any remaining concerns as you finalize your assessment.

Reviewer_AhpXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether reinforcement learning with verified training data and task-specific rewards can improve the text-to-SQL capabilities of large language models. The authors introduce BIRD-Platinum, a dataset of approximately 2,500 instances sampled from BIRD Train and curated through an LLM-assisted, multi-round expert verification process. They report correcting annotation errors in 61.1% of the instances, including errors in SQL queries, natural-language questions, and external knowledge.The paper further proposes ReViSQL, a reward-shaping approach that combines execution-based rewards with bounded SQL equivalence verification using VeriEQL and process penalties designed to encourage explicit use of external knowledge. The authors fine-tune Kimi-K2.6 and Qwen3-235B-A22B using the curated dataset. ReViSQL-K2.6 achieves 93.0% and 94.2% execution accuracy on Arcwise-Plat-SQL and Arcwise-Plat-Full, respectively, using self-consistency over 16 candidates.

Critical strengths
  1. The construction of BIRD-Platinum represents a substantial data curation effort. The authors manually verify approximately 2,500 instances through multiple rounds of expert review and report annotation errors in 61.1% of the instances, including 52.1% with incorrect SQL queries. These findings highlight the severity of annotation noise in existing text-to-SQL datasets and provide a potentially valuable resource for future research. The observed improvements from training on the corrected data further demonstrate the practical importance of data quality for reinforcement learning.
  2. The paper reports impressive empirical results, with ReViSQL-K2.6 achieving 93.0% and 94.2% accuracy on Arcwise-Plat-SQL and Arcwise-Plat-Full.
Critical weaknesses
  1. The data verification pipeline mainly follows established practices, while ReViSQL combines an existing SQL equivalence verifier with heuristic reward penalties. In particular, the process reward checks the presence of predefined reasoning blocks rather than whether external knowledge is correctly applied.
  2. The paper does not separately ablate the verification-aware outcome reward and the process reward. Moreover, the effect of the process reward is not distinguished from that of the additional prompting instructions.
Questions for authors
  1. Can the authors provide separate ablation results for the verification-aware outcome reward and the process reward? In particular, can the effect of process rewards be distinguished from that of the additional prompting instructions?
  2. Since the process reward relies on string-level checks, how do the authors ensure that the generated reasoning blocks reflect correct use of external knowledge rather than mere compliance with the required format?
4 comments

We thank the reviewer for their insightful feedback. We concisely addressed them below and accordingly revised the paper with changes highlighted in blue.

W1.2 / Q2: Since the process reward relies on string-level checks, how do the authors ensure that the generated reasoning blocks reflect correct use of external knowledge rather than mere compliance with the required format?

We acknowledge that the string check we used in the process reward penalty does not verify that the external knowledge is used correctly. We show that such string checks are effective based on our analysis on the base model behavior, training dynamics, and pilot experiments. In the revised paper, we added this discussion to Section 4.2. We summarize the key points here.

Using required reasoning blocks is a strong proxy for correct external knowledge use. Before training, we evaluated Kimi-K2.6 on the validation set (Appendix E). Among the model responses that contain the required reasoning blocks, it correctly uses the external knowledge for 97.8% of the time. In contrast, among the responses that do not contain the blocks, it only correctly uses the external knowledge for 77.6% of the time.

Requiring reasoning blocks is also effective during training. In each of the eight training epochs, we sample 100 queries that receive full rewards (no process penalty) and label them with an LLM judge that we separately tested (Appendix I.1). We find that 95–99% of the full-reward queries satisfy the external knowledge. Hence, our reward rarely gives full credit to queries that misuse the external knowledge.

Checking correct use semantically results in similar accuracy at higher cost. In a pilot training experiment with Nemotron 3.5 Lightning using Tinker, we replaced the string check with an LLM judge (GPT 6 Luna) of external-knowledge use, following Reasoning-SQL [3]. We find that the model trained with the LLM judge achieved a validation accuracy of 84.2%, matching the model trained with string checks (83.4%). However, the LLM judge incentivized the model to produce 1.5x longer responses, potentially due to the judge's verbosity bias [4, 6], costing 1.3x more to train.

W1.1: The data verification pipeline mainly follows established practices, while ReViSQL combines an existing SQL equivalence verifier with heuristic reward penalties.

In the revised paper, we clarified the novelty of this work in terms of data verification and RL reward design in Section 2 and ran additional experiments to compare ReViSQL with prior RL rewards in Section 5.2. We summarize the key points here.

Data verification. To our knowledge, no prior text-to-SQL verification pipeline corrects annotation errors through multiple rounds of independent correction and verification among human experts.

  • Qualitative comparison: Prior data examination pipelines filter or detect errors without correcting them [3, 7, 9, 10, 11]. Wretblad et al. [8] correct annotation errors manually without a reusable pipeline. SAPAR [12] iterates between an LLM reviewer and human annotators. In our pipeline, we used multi-turn correction and verification among multiple experts, with the LLM reviewer only for initial flagging.

  • Quantitative evidence: As shown in Figure 3a, the LLM reviewer from prior work [12] detects only 24.5% of the errors we corrected.

Reward design. To our knowledge, ReViSQL is the first text-to-SQL RL method to use formal verification in rewards and use process penalties against ignoring external knowledge. We ran additional experiments to compare ReViSQL with prior RL rewards for text-to-SQL [3, 13, 14, 15] by training Nemotron 3.5 Lightning via Tinker. ReViSQL outperforms prior rewards by 0.6-5.2%, 2.8-3.0%, 6.4-6.8%, and 7.4-10.8% on Arcwise-Plat benchmarks, respectively.

W2 / Q1: Can the authors provide separate ablation results for the verification-aware outcome reward and the process reward? In particular, can the effect of process rewards be distinguished from that of the additional prompting instructions?

We ran an additional ablation study to isolate the effect of the two reward components. We find that neither component alone fully accounts for the gain of ReViSQL-K2.6. In the revised paper, we added the new results in Section 5.3. We summarize the key findings as follows.

Structured promptVeriEQLProcessArcwise-Plat-FullArcwise-Plat-SQL
No89.9688.55
YesYes90.56 (+0.60)88.76 (+0.21)
YesYes91.16 (+1.20)90.16 (+1.61)
YesYesYes93.78 (+3.82)91.37 (+2.82)

Separate effects. Compared to the standard execution-based outcome reward, process penalties alone only slightly improve accuracy (+ 0.2-0.6%), while verification-aware outcome reward alone improves it by 1.2–1.6%. Neither of them individually recovers the gain of ReViSQL-K2.6 over the standard reward.

Process penalties and prompting instructions. We show that with the same structured prompt and the same verification-aware reward, adding the penalties improves accuracy by 1.2–2.6%. Without the process penalties, the fine-tuned model violates the external-knowledge verification requirement 3.8x as often as ReViSQL. Through our manual review, we also find that the share of its incorrect queries that misuse the external knowledge is 3.6x that of ReViSQL. Therefore, the prompt alone does not ensure that the model performs the required verification, while the process penalties do.


[1] Wang, Yiping, et al. "Reinforcement learning for reasoning in large language models with one training example." NeurIPS (2025)

[2] Xin, Rihui, et al. "Surrogate signals from format and length: Reinforcement learning for solving mathematical problems without ground truth answers." arXiv:2505.19439 (2025).

[3] Pourreza, Mohammadreza, et al. "Reasoning-sql: Reinforcement learning with sql tailored partial rewards for reasoning-enhanced text-to-sql." arXiv:2503.23157 (2025).

[4] Zheng, Lianmin, et al. "Judging llm-as-a-judge with mt-bench and chatbot arena." NeurIPS (2023).

[5] Wang, Peiyi, et al. "Large language models are not fair evaluators." ACL (2024).

[6] Wu, Minghao, and Alham Fikri Aji. "Style over substance: Evaluation biases for large language models." COLING (2025).

[7] Pourreza and Rafiei. "Evaluating Cross-Domain Text-to-SQL Models and Benchmarks." EMNLP (2023).

[8] Wretblad et al. "Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark." ACL (2024).

[9] Kim et al. "FLEX: Expert-level False-Less EXecution Metric for Text-to-SQL Benchmark." NAACL (2025).

[10] Klopfenstein et al. "SpotIt: Evaluating Text-to-SQL Evaluation with Formal Verification." ICLR (2026).

[11] Liu et al. "NL2SQL-BUGs: A Benchmark for Detecting Semantic Errors in NL2SQL Translation." KDD (2025).

[12] Jin et al. "Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards." PVLDB, (2026).

[13] Lee, Jaehoon, et al. "EXPO-SQL: Execution-based Clause-level Policy Optimization for Text-to-SQL." Findings of the Association for Computational Linguistics: ACL (2026).

[14] Zhang, Shihao, et al. "Progress-SQL: Improving Reinforcement Learning for Text-to-SQL via Progressive Rewards." arXiv:2606.06825 (2026).

[15] Dai, Yaxun, et al. "Every Step Counts: Step-Level Credit Assignment for Tool-Integrated Text-to-SQL." ACL (2026).

Dear Reviewer AhpX,

Thank you again for your feedback. In the revised paper, we added reward-component ablations with a fixed-prompt comparison (Section 5.3), an analysis of the reasoning-block checks, a pilot comparison with an LLM judge (Section 4.2), and comparisons with four prior RL reward methods (Section 5.2).

Please let us know whether these updates address your concerns. We would be happy to address any remaining questions.

Best,

The Authors

Thank you for the detailed responses and additional experimental results. I have no further questions at this stage.

Reviewer_K5UsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BIRD-Platinum, an expert-verified Text-to-SQL training dataset, and ReViSQL, an RL reward shaping method that combines SQL equivalence verification with process rewards for external knowledge use. Experiments show that verified training data significantly improves performance across BIRD and Spider2 benchmarks. Fine-tuning Kimi-K2.6 with ReViSQL achieves human-level accuracy on expert-verified BIRD benchmarks and outperforms existing Text-to-SQL systems at lower inference cost.

Critical strengths
  1. The paper introduces BIRD-Platinum, a dataset of 2.5k Text-to-SQL instances verified through multiple rounds of expert review. The authors find annotation errors in 61.1% of the instances, highlighting a significant problem with existing training data. The experiments in Figure 8 show that training on BIRD-Platinum consistently improves accuracy by 11–16 percentage points across Arcwise-Plat and Spider2 benchmarks compared with training on the original BIRD data. This is a valuable contribution to improving Text-to-SQL training.
  2. The paper identifies two important limitations of standard execution-based rewards: false positive rewards for semantically incorrect SQL and insufficient incentives to use external knowledge. The proposed ReViSQL method addresses these issues through SQL equivalence verification and process rewards. Figure 7 shows that reward shaping provides an additional 2.8–3.8 percentage points of improvement over standard rewards. The final model also achieves competitive human-level accuracy on Arcwise-Plat and outperforms several strong Text-to-SQL systems, demonstrating the practical effectiveness of the proposed approach.
Critical weaknesses
  1. In Section 5, the authors compare ReViSQL with several strong Text-to-SQL pipelines and open-weight models. However, the paper lacks direct comparisons with closely related RL-based Text-to-SQL training methods, particularly those focusing on task-specific reward design. Without such comparisons under similar training settings, it is difficult to determine whether ReViSQL provides meaningful improvements over existing RL approaches rather than only over standard execution-based rewards.
  2. In Section 4.2, ReViSQL combines bounded SQL equivalence verification with process rewards for external knowledge use. While Figure 7 shows that the complete reward design improves accuracy, the paper does not separately evaluate the contributions of these two components. This makes it difficult to identify which component drives the improvements. In addition, the human-level claim in Section 5.2 relies on a 92.96% human performance reference from the original BIRD benchmark rather than a direct human evaluation on Arcwise-Plat. This limits how strongly the reported results support the claim of human-level performance.
Questions for authors
  1. In Section 5.1, the authors compare ReViSQL against strong Text-to-SQL pipelines and open-weight models, but provide limited direct comparisons with recent RL-based training methods. Several closely related works have explored SQL-specific reward design, including Reasoning-SQL [1], EXPO-SQL [2], Progress-SQL [3], and FineStep [4]. These methods use different forms of structural, execution-based, or step-level feedback to improve SQL reasoning. Could the authors clarify how ReViSQL differs from these reward designs and why these methods were not included in the experimental comparisons? In particular, would ReViSQL still provide meaningful improvements over these approaches under comparable training settings? This would help assess the novelty and effectiveness of the proposed reward shaping method.
  2. In Section 4.2, Equation (1), ReViSQL combines a verification-aware outcome reward based on VeriEQL with process penalties that encourage external-knowledge use. However, Figure 7 only compares the complete ReViSQL reward against standard execution-based rewards. Could the authors clarify how much of the reported 2.8–3.8 percentage-point improvement comes from each component? In particular, is the main improvement driven by reducing false positive execution rewards or by encouraging explicit external-knowledge verification?
  3. In Section 5.2, the authors claim that ReViSQL-K2.6 achieves human-level Text-to-SQL performance by comparing its results on Arcwise-Plat-Full and Arcwise-Plat-SQL against the 92.96% human performance reference reported for the original BIRD benchmark. However, these are different evaluation settings, and Arcwise-Plat includes expert corrections to the original benchmark annotations. Could the authors clarify why the original BIRD human performance estimate is an appropriate reference for these corrected benchmarks? Is there any evidence that the human performance level remains comparable after these corrections?

[1] Pourreza, Mohammadreza, et al. "Reasoning-sql: Reinforcement learning with sql tailored partial rewards for reasoning-enhanced text-to-sql." arXiv preprint arXiv:2503.23157 (2025). [2] Lee, Jaehoon, et al. "EXPO-SQL: Execution-based Clause-level Policy Optimization for Text-to-SQL." Findings of the Association for Computational Linguistics: ACL 2026. 2026. [3] Zhang, Shihao, et al. "Progress-SQL: Improving Reinforcement Learning for Text-to-SQL via Progressive Rewards." arXiv preprint arXiv:2606.06825 (2026). [4] Dai, Yaxun, et al. "Every Step Counts: Step-Level Credit Assignment for Tool-Integrated Text-to-SQL." arXiv preprint arXiv:2605.04719 (2026).

2 comments

We thank the reviewer for their comments. We addressed them below and revised the paper with changes highlighted in blue.

W1/Q1: direct comparisons with recent RL-based training methods

We agree that these four methods are relevant baselines. In the revised paper, we added qualitative analysis of them in Section 2.1 and quantitatively compared ReViSQL with them in Section 5.2. We summarize the key points here.

How ReViSQL differs. We find that all four methods still decide whether a query is correct by execution-based grading and just add finer-grained signals around it:

  • Reasoning-SQL [1] applies an LLM judge to queries with incorrect results.
  • EXPO-SQL [2] assigns clause-level partial rewards to queries with incorrect results.
  • Progress-SQL [3] and FineStep [4] reward intermediate steps.

Therefore, they do not explicitly penalize queries that coincidentally return the gold result, including queries that ignore the external knowledge. In contrast, ReViSQL addresses these failures directly.

Quantitative comparison. We fine-tuned Nemotron 3.5 Lightning on BIRD-Platinum with each reward using Tinker. We keep all other training settings consistent (Appendices E and F.3). As shown in the following table, ReViSQL outperforms all four rewards on both benchmarks: Reasoning-SQL by 0.6–5.2%, EXPO-SQL by 2.8–3.0%, Progress-SQL by 6.4–6.8%, and FineStep by 7.4–10.8%.

RewardArcwise-Plat-Full (%)Arcwise-Plat-SQL (%)
FineStep69.8870.48
Progress-SQL73.9071.49
EXPO-SQL77.9174.90
Reasoning-SQL80.1272.69
ReViSQL80.7277.91

W2.1 / Q2: Could the authors clarify how much of the reported 2.8–3.8 percentage-point improvement comes from each component? In particular, is the main improvement driven by reducing false positive execution rewards or by encouraging explicit external-knowledge verification?

We ran an additional ablation study to isolate the effect of the two reward components. We find that neither component alone accounts for the gain of ReViSQL-K2.6. The main improvement is driven by both reducing false positive rewards and encouraging explicit external-knowledge verification In the revised paper, we added the new results in Section 5.3. We summarize the key findings as follows.

VeriEQLProcessArcwise-Plat-FullArcwise-Plat-SQL
89.9688.55
Yes90.56 (+0.60)88.76 (+0.21)
Yes91.16 (+1.20)90.16 (+1.61)
YesYes93.78 (+3.82)91.37 (+2.82)

Compared to the standard execution-based outcome reward, process penalties alone only slightly improve accuracy (+ 0.2-0.6%), while verification-aware outcome reward alone improves it by 1.2–1.6%. Neither of them individually recovers the gain of ReViSQL-K2.6 over the standard reward.

We find that the two components address complementary failures. Even though the model trained with only verification-aware reward uses the same prompt as ReViSQL, it violates the external-knowledge verification requirement 3.8x as often as ReViSQL. Through our manual review, we also find that the share of its incorrect queries that misuse the external knowledge is 3.6x that of ReViSQL. Therefore, both components are necessary.

W2.2 / Q3: Could the authors clarify why the original BIRD human performance estimate is an appropriate reference for these corrected benchmarks?

We acknowledge that achieving the 92.96% reference does not support our claim of achieving human-level performance, since the reference was measured on the BIRD Test set, which differs from Arcwise-Plat. Therefore, we removed the human-level claim throughout the revised paper.

Nevertheless, our results demonstrate substantial accuracy gains and lower inference costs. ReViSQL-K2.6 outperforms OpenSearch, the strongest prior open-source pipeline, by 8.4% on Arcwise-Plat-SQL and 5.6% on Arcwise-Plat-Full, while reducing per-query inference cost by 37%. Compared with GPT-5.6 Sol Ultra and Claude Fable 5, it achieves 3.2–7.9% higher accuracy on Arcwise-Plat-SQL and 1.0–3.8% higher accuracy on Arcwise-Plat-Full, using only 12–15% of their inference cost.


[1] Pourreza, Mohammadreza, et al. "Reasoning-sql: Reinforcement learning with sql tailored partial rewards for reasoning-enhanced text-to-sql." arXiv:2503.23157 (2025).

[2] Lee, Jaehoon, et al. "EXPO-SQL: Execution-based Clause-level Policy Optimization for Text-to-SQL." Findings of the Association for Computational Linguistics: ACL (2026).

[3] Zhang, Shihao, et al. "Progress-SQL: Improving Reinforcement Learning for Text-to-SQL via Progressive Rewards." arXiv:2606.06825 (2026).

[4] Dai, Yaxun, et al. "Every Step Counts: Step-Level Credit Assignment for Tool-Integrated Text-to-SQL." ACL (2026).

Dear Reviewer K5Us,

Thank you again for your feedback. In the revised paper, we compare ReViSQL with four RL reward baselines under identical training settings and find that it outperforms all four on both benchmarks (Section 5.2). We also added an ablation showing that the two reward components work best together (Section 5.3) and removed the human-level performance claim.

Please let us know if you have any remaining concerns. We’d be happy to address them.

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.