- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces BIRD-Platinum, a roughly 2,500-example subset of BIRD Train corrected through independent expert review, and REVISQL, an RL reward that adds bounded SQL-equivalence checks and external-knowledge process penalties to execution grading. A same-backbone Qwen comparison reports gains from training on BIRD-Platinum across Arcwise-Plat and Spider2. The full Kimi-K2.6 system reaches 93.8% with greedy decoding on Arcwise-Plat-Full and 93.0% with 16-sample self-consistency on Arcwise-Plat-SQL. The paper compares these scores with a previously reported 92.96% human proxy on BIRD.
Critical strengths
- The data effort is substantive: independent experts corrected gold SQL in 52.1% of examples, found some annotation error in 61.1%, and rejected 16.2% of first-round proposed corrections on verification. The same-backbone training comparison also shows gains beyond BIRD on Spider2, making this more than a benchmark-specific cleanup exercise.
- The paper diagnoses a real weakness in execution-based RL rewards. Bounded verification refutes equivalence for an average of 32.8% of execution-accepted rollouts in a pilot run, and the complete REVISQL intervention adds 2.8–3.8 points over standard rewards when both train on BIRD-Platinum. This is useful evidence for improving the training signal, even though the individual reward components are not separately identified.
Critical weaknesses
- The gains are better established for the combined interventions than for their proposed causes. BIRD-Platinum is a sampled subset of BIRD Train, but the data comparison does not report a control trained on the original annotations of those same examples with the same rollout and update budget. The large gain may therefore mix annotation correction with example selection or training-set size. Likewise, the REVISQL ablation adds bounded verification, a structured prompt, and string-checked process penalties together. The small manual audit of external-knowledge failures is encouraging, but it does not separate those effects or measure semantic compliance across rewarded SQL outputs. The paper supports the value of the packages more clearly than the causal account of each ingredient.
- “Human-level” is a comparison across different evaluation conditions. The 92.96% human proxy was reported for BIRD, whereas the model is scored on Arcwise-Plat variants with corrected gold SQL and, in the Full variant, corrected questions and external knowledge. Those repairs improve the evaluation, but they also make the historical number an unsuitable parity threshold without a matched human assessment. The model's strong scores stand on their own; they do not yet establish the title's human-level claim.
Questions for authors
- From the existing runs, were the BIRD Train and BIRD-Platinum comparisons matched on example IDs and RL rollout/update budget? For REVISQL, what fraction of VeriEQL-refuted or inconclusive rollouts still received positive training advantage, and what evidence shows that high-reward SQL actually satisfies the supplied external knowledge? These results would clarify whether the reported gains support the proposed mechanisms rather than only the combined setup.
- The 92.96% human proxy was measured on BIRD, whereas the model is evaluated on corrected variants of that benchmark. What were the proxy's exact instances, input information, and scoring rule? Since the corrected variants change the gold SQL, and the Full variant also changes the questions and external knowledge, what evidence establishes the historical proxy as a valid human comparator? Numerical proximity between aggregate scores alone does not establish parity.
4 comments
Q2: The 92.96% human proxy was measured on BIRD, whereas the model is evaluated on corrected variants of that benchmark.
We acknowledge that the historical proxy is not a valid human comparator on Arcwise-Plat benchmarks. The BIRD Team measured the 92.96% execution accuracy on the BIRD Test split, which contains different data instances from Arcwise-Plat. Therefore, we removed the human-level claim throughout the revised paper.
Nevertheless, our results demonstrate substantial accuracy gains and lower inference costs. ReViSQL-K2.6 outperforms OpenSearch, the strongest prior open-source pipeline, by 8.4% on Arcwise-Plat-SQL and 5.6% on Arcwise-Plat-Full, while reducing per-query inference cost by 37%. Compared with GPT-5.6 Sol Ultra and Claude Fable 5, it achieves 3.2–7.9% higher accuracy on Arcwise-Plat-SQL and 1.0–3.8% higher accuracy on Arcwise-Plat-Full, using only 12–15% of their inference cost.
Thank you for the additional experiments and clarifications. I have no further questions at this point.
Thank you for your thoughtful feedback and for considering our additional experiments and clarifications. We are glad to have addressed your questions. We would also be happy to clarify any remaining concerns as you finalize your assessment.
We thank the reviewer for their comments. We addressed them below and revised the paper with changes highlighted in blue.
We clarify that the existing BIRD Train and BIRD-Platinum comparison uses the same RL rollout budget and maximum update budget. However, they do not match on example IDs. Example IDs of BIRD-Platinum is a random sample of that of BIRD-Train.
We additionally trained Kimi K2.6 on BIRD-Train-Match, which matches BIRD-Platinum on example IDs with the original annotation from BIRD Train. We kept other settings consistent. BIRD-Train-Match achieved 78.31% accuracy on Arcwise-Plat-Full and 72.89% on Arcwise-Plat-SQL, underperforming the full BIRD Train by 1.4-1.6% and BIRD-Platinum by 15.5-18.5%. Therefore, the gain of BIRD-Platinum comes from correcting the annotations, not from example selection or training-set size.
We added these results to Section 5.3 of the revised paper.
We ran an additional ablation study to isolate the effect of the two reward components. We find that neither component alone accounts for the gain of ReViSQL-K2.6. In the revised paper, we added the new results in Section 5.3. We summarize the key findings as follows.
Compared to the standard execution-based outcome reward, process penalties alone only slightly improve accuracy (+ 0.2-0.6%), while verification-aware outcome reward alone improves it by 1.2–1.6%. Neither of them individually recovers the gain of ReViSQL-K2.6 over the standard reward.
We find that the two components address complementary failures. Even though the model trained with only verification-aware reward uses the same prompt as ReViSQL, it violates the external-knowledge verification requirement 3.8x as often as ReViSQL. Through our manual review, we also find that the share of its incorrect queries that misuse the external knowledge is 3.6x that of ReViSQL. Therefore, both components are necessary.
We analyzed the generated SQL queries and their rewards over the training of ReViSQL. We find that the fraction of VeriEQL-refuted rollouts that still receive a positive advantage drops from 39% to 5% over training, while 95–99% of high-reward queries satisfy the external knowledge. In the revised paper, we added these analyses to Section 4.2. We summarize the key findings as follows.
VeriEQL-refuted and inconclusive rollouts. We find that the fraction of VeriEQL-refuted rollouts that still receive a positive advantage decreases from 39% in the first epoch to 5% in the eighth epoch. This trend indicates that the model generates more semantically correct queries. Furthermore, ReViSQL does not penalize inconclusive verdicts, because they reflect VeriEQL's internal incompleteness. Nevertheless, the fraction of inconclusive rollouts of all rollouts stays stable throughout training, fluctuating between 5.5% and 10.1%.
External-knowledge compliance of high-reward queries. In each of the eight training epochs, we randomly sampled 100 queries that received full reward and labeled them with an LLM judge (Appendix I.1). In every epoch, we find that 95–99% of the sampled queries satisfy the external knowledge. Therefore, our reward rarely gives full credit to SQL that violates the external knowledge.