acceptodds
Under review as a conference paper at ICLR 2027

Residual Skill Optimization for Text-to-SQL Ensembles

Abstract

Text-to-SQL ensembles improve over single-candidate generation by drawing multiple SQL candidates and selecting one, but their effectiveness is bounded by Pass@K, the probability that at least one of K candidates is correct. Existing methods source diversity heuristically through stochastic decoding or prompt variants, leaving candidate sets dominated by correlated failures. We present DivSkill-SQL, a residual skill optimization framework that builds complementary agentic Text-to-SQL ensembles without model fine-tuning: each new skill is optimized on examples the current skill ensemble fails on, provably targeting its marginal contribution to Pass@K. On Spider2-Lite, DivSkill-SQL improves selected accuracy by up to +11.1 points on Snowflake and +8.3 on BigQuery over the strongest ensemble baseline, with consistent gains across two base models (Opus-4.6 and GPT-5.4). Skills optimized on a single dialect transfer without retraining across dialects (Snowflake, BigQuery, SQLite) and to a different task formulation, such as BIRD-Critic (2.6 pts). Error diagnostics show up to 3x fewer hallucinated schema references and function calls, indicating that gains come from genuinely reliable complementary skills rather than surface-form variation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Text2SQL, Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_F1fnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose an ensemble method for text-to-sql generation, where prompts are constructed such that it addresses the residual failures of the ensemble. By constructing their prompts this way, they are able to build a diverse set of prompts suitable for ensemble. During inference, they apply an LLM judge to select the right prompt for the right problem.

Critical strengths
  1. The authors achieve significant improvements over other methods.
  2. The authors propose an intuitive method to for text-to-sql ensembles.
Critical weaknesses
  1. The proposed method is compute intensive. In addition to running multiple prompts, the authors require additional pairwise LLM judges during inference, making it very costly.
  2. Experiments. The authors do not analyse the effect of number of ensemble components.
Questions for authors

How much time does the LLM judges incur?

2 comments

We thank the reviewer for the helpful feedback, and address each of the concerns below. All changes in the revised paper are highlighted in blue for convenience of reading (we will change it back after the rebuttal ends).

W1 & Q1: The method is compute-intensive. It runs multiple prompts and adds pairwise LLM judges at inference. How much time do the LLM judges incur?

We thank the reviewer for raising this concern and bringing in this dimension. We agree that pairwise judging grows quadratically with the number of candidates and can become expensive at large K. However, in practice DivSkill-SQL does not need to scale beyond K = 8, and a much smaller ensemble is often enough: as we later show in W2, with only K = 2 or K = 4 candidates DivSkill-SQL already reaches decent performance. Moreover, the pairwise comparisons for the same question share a large amount of context, which is served as cached input. As shown in the cost breakdown below, more than half of the pairwise judge's input tokens are cached, which costs 90% less than standard (uncached) input tokens. This substantially reduces the actual cost of pairwise judging.

We would also like to clarify that we adopt the pairwise judge from CHASE-SQL to reach higher final quality, and this is a replaceable component of our system. To better understand quality-cost trade-off of the judge, we additionally compare with a one-shot judge that compares all distinct candidates in a single LLM call, and majority voting. The pairwise judge leverages parallelism due to independence between comparisons, and it runs with at most 8 concurrent processes (aligned with learning stage). We report the accuracy, token cost breakdown and time below:

SelectorSQLiteSnowflakeBigQueryCached Input TokensUncached Input TokensOutput TokensCost ($)Time (s)
Majority vote65.9357.0060.9800000
One-shot71.1158.9462.4404.4K3770.0175.7
Pairwise71.8561.8463.4131.7K27.9K7.5K0.19012.9

The results indicate that one-shot judge could be a decent low-budget alternative. Importantly, even with this cheap judge, DivSkill-SQL still outperforms CHASE-SQL with the pairwise judge on all three dialects. Users can therefore choose the selector according to their budget.

We have added the above results and analysis to the revised paper (Section 4.3, "Choice of selector").

W2: The effect of the number of ensemble components is not analyzed.

We thank the reviewer for the suggestion, and conducted an ablation on the number of candidates k:

MethodMetrick=1k=2k=4k=8
CHASE-SQLPass@k61.5770.3778.8983.70
CHASE-SQLSelected61.5763.3365.1965.19
DivSkill-SQLPass@k62.8773.5281.1184.44
DivSkill-SQLSelected62.8767.9668.1571.11

DivSkill-SQL achieves higher Pass@k and selected accuracy than CHASE-SQL for every k, and with only two candidates it already outperforms CHASE-SQL with eight in selected accuracy.

We have added the results and analysis to the revised paper (Section 4.3, "Number of ensemble components").

We sincerely appreciate your time and valuable feedback, which has helped us improve our work.

With the discussion period ending in a day, we look forward to your thoughts on our response. Please let us know if there is anything more we can do to address your comments!

Thanks again for your time and feedback!

Reviewer_LMh4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DIVSKILL-SQL, which addresses the problem that current Text-to-SQL self-consistency voting lacks genuinely new reasoning paths, through residual skill optimization. By making the complementarity of candidate SQL queries an explicit optimization objective, and by iteratively optimizing skill prompts on erroneous examples across multiple batches, it achieves complementarity between skill capabilities and the generated SQL. Experiments demonstrate that DIVSKILL-SQL yields performance gains over existing baselines, while also exhibiting good cross-dialect generalization and a low redundancy rate in tool-use trajectories.

Critical strengths
  1. Well-grounded motivation: Voting mechanisms in Text-to-SQL often rely on stochastic decoding and input or workflow variants. DIVSKILL-SQL's design of optimizing complementarity among skills is thus well motivated and valuable.

  2. Clear methodology: Through multiple batches and skill-order rotation, it iteratively optimizes skill prompts on the residual failure examples produced by each skill-conditioned agent, thereby improving complementarity among different skills by construction.

  3. Beyond comparing Pass@k and selected accuracy, it also analyzes the similarity between agent-generated trajectories and compares accuracy across SQL dialects.

Critical weaknesses
  1. Limited contribution. The core innovation of the paper lies in optimizing a skill on the failure examples of other skills to increase its marginal contribution to Pass@K. This is essentially an optimization of the candidate generation procedure for Text-to-SQL, rather than an improvement of the key task-specific problems in Text-to-SQL.

  2. The problems in the selection stage remain unresolved. DIVSKILL-SQL still exhibits a large gap between Pass@8 and selected accuracy (Sel. Acc.).

  3. DIVSKILL-SQL employs several methods to improve the cross-dialect transferability of skills—for example, instructing the reflector to avoid dialect-specific syntax in the prompt. This reduces the risk of dialect overfitting by construction, but its connection to the paper's core residual skill optimization is relatively weak.

Questions for authors
  1. The Spider2.0 dataset primarily targets end-to-end SQL generation, whereas BIRD-Critic primarily targets SQL debugging and repair. The paper uses BIRD-Critic as transferability evidence. On BIRD-Critic, the skills of DIVSKILL-SQL are trained on BIRD-mini-dev, while BIRD-Critic is a debugging task. Therefore, the gains on BIRD-Critic may partly stem from the transfer of general SQL capability rather than from residual skill complementarity, the core mechanism. Suggest adding a control: directly running residual skill optimization on BIRD-Critic and comparing it with the result of "training on BIRD-mini-dev and transferring to BIRD-Critic."

  2. When a weaker selector is used (e.g., without LLM-based pairwise judging), are the Pass@k gains brought by DIVSKILL-SQL retained in the final selected SQL?

3 comments

We thank the reviewer for the constructive and insightful feedback, and appreciate the positive assessment of our motivation and methodology. We address each of the concerns below. All changes in the revised paper are highlighted in blue for convenience of reading (we will change it back after the rebuttal ends).

W1: Limited contribution. The method optimizes the candidate generation procedure rather than the key task-specific problems in Text-to-SQL.

Thank you for your feedback. We think better candidate generation is a central line of Text-to-SQL work. Existing works such as CHASE-SQL [1] and XiYan-SQL [2] all try to generate diverse candidates and select among them, which substantially outperforms a single generation. We identify and address a key obstacle along this path: candidates from existing methods tend to fail on the same questions, and so adding more candidates helps little.

We agree that Text-to-SQL-specific priors, such as dialect-specific rules or schema-linking modules, could push performance further, e.g., by saving agent turns spent on easy syntax errors and schema mismatches. We have added this discussion to the future work in the revised paper (Section 5).

W2: The problems in the selection stage remain unresolved. There is still a large gap between Pass@8 and selected accuracy.

We agree that selection remains an open problem shared by all ensemble methods, and an important future direction. Besides the difficulty of building a strong selector, we observe that around half of the gap comes from ambiguous gold labels, where the selected SQL returns an admissible result that is still marked as wrong, a finding shared by [3]. For example, for a question asking for the cost of every product, we return all product IDs with their costs, but the gold table also includes the product names, which the question does not ask for.

We have added the above discussion to the revised paper (Section 5) to enhance the understanding of the gap.

W3: Several design choices for cross-dialect transfer (e.g., instructing the reflector to avoid dialect-specific syntax) are only weakly connected to residual skill optimization.

Thank you for the feedback. We agree that the transferability stems from the mentioned design choices, which pushes the skills to handle high-level logics. And residual optimization forces the learned skills to cover more general strategies to make sure the learned transferable skills are not redundant. We have added this point to the revised paper (Section B).

Q1: Gains on BIRD-Critic may come from transferring general SQL capability rather than from residual complementarity. Suggest running residual optimization directly on BIRD-Critic.

We thank the reviewer for sharing this sharp insight. We agree that the skills learned by DivSkill-SQL mainly improve general end-to-end SQL generation, which carries over to the SQL-debugging tasks in BIRD-Critic even though the skills were optimized on standard Text-to-SQL data. Following the suggestion, to better understand how much further can task-setting-specific data bring, we added a "pseudo performance ceiling" that directly optimizes skills on in-distribution BIRD-Critic:

Methodpass@1pass@8Sel. acc.
DIN-SQL30.38––
ReFoRCE41.50–43.21
CHASE-SQL44.6952.2646.23
DivSkill-SQL (transfer)46.1654.5348.87
DivSkill-SQL (direct)48.1156.6050.75

Optimizing directly on in-distribution BIRD-Critic data further improves selected accuracy by 1.88 points over the transferred skills. This confirms that in-distribution, debugging-style data further helps, on top of the general SQL generation ability learned from standard Text-to-SQL data. We have added the above results and discussion to the revised paper (Section 4.2 "Transfer across SQL dialects and task settings", and Table 3).

[1] Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan Arik. Chase-sql: Multi-path reasoning and preference optimized candidate selection in text-to-sql. ICLR 2025 [2] Yifu Liu, Yin Zhu, Yingqi Gao, Zhiling Luo, Xiaoxia Li, Xiaorong Shi, Yuntao Hong et al. Xiyan-sql: A novel multi-generator framework for text-to-sql. TKDE 2026 [3] Tengjun Jin, Yoojin Choi, Yuxuan Zhu, and Daniel Kang. Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards. VLDB 2026

Q2: With a weaker selector (e.g., without LLM-based pairwise judging), are the Pass@k gains retained in the final selected SQL?

Thank you for the question. We have conducted an experiment with various judge choices to answer this.

SelectorMethodSQLiteSnowflakeBigQuery
Majority voteCHASE-SQL66.6755.5657.07
Majority voteDivSkill-SQL65.9357.0060.98
One-shotCHASE-SQL65.1957.4958.05
One-shotDivSkill-SQL71.1158.9462.44
PairwiseCHASE-SQL66.6757.9759.51
PairwiseDivSkill-SQL71.8561.8463.41

As shown in the table above, DivSkill-SQL keeps its advantage over CHASE-SQL with majority voting and with a one-shot judge. Importantly, DivSkill-SQL with the one-shot judge outperforms CHASE-SQL with the pairwise judge on all three dialects. This indicates that the candidate SQL pool generated by DivSkill-SQL remains strong under cheap selection.

We have added the above results and analysis to the revised paper (Section 4.3, "Choice of selector").

We sincerely appreciate your time and valuable feedback, which has helped us improve our work.

With the discussion period ending in a day, we look forward to your thoughts on our response. Please let us know if there is anything more we can do to address your comments!

Thanks again for your time and feedback!

Reviewer_uhnTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a prompt optimization scheme for Text-to-SQL task. The method start with a base set of skills and iteratively optimizes prompts on examples unresolved by previous skills to improve overall Pass@K. At inference time, a group of skill-conditioned agents generate candidates independently and a pairwise selector chooses the final query. Experiments report improvements across multiple SQL dialects, model backbones, and a SQL-debugging task. The results show accuracy improvements compared to DIN-SQL, CHASE-SQL, and REFORCE.

Critical strengths

The paper proposes a training free method that empirically improves the text-to-sql accuracy over the agentic text-to-sql baselines, including CHASE-SQL with 2 frontier models.

Critical weaknesses
  • The novelty of this paper needs further justification as earlier methods already construct prompt ensembles using difficult examples or reflective failure feedback. The contribution appears to be incremental compared to "Boosted Prompt Ensembles for Large Language Models".

  • The contribution of residual optimization is not well or clearly ablated. Therefore, the mechanism behind the gains compared to other prompt optimization methods, such as GEPA, remains unclear.

Questions for authors
  • Could you clarify how Opt. Base is constructed? In particular, does it retain the same eight diverse seed prompts as DIVSKILL-SQL? If not, please compare against independently optimizing those same seeds without residual filtering, holding the optimization and inference budgets fixed.

  • What are the actual inference costs and end-to-end latencies compared to other baselines, including generation, tool use, and selection? What is the separate cost of learning the skill bank?

  • Can authors add an adapted Boosted Prompt Ensembles baseline to help to establish the additional value of residual-conditioned instruction refinement?

  • How do the learned skills generalize to unseen schemas and query patterns, and what failure modes explain cases where optimization reduces test accuracy compared with the original seed skills?

3 comments

We thank the reviewer for the detailed and helpful feedback, and address each of the concerns below. All changes in the revised paper are highlighted in blue for convenience of reading (we will change it back after the rebuttal ends).

W1 & Q3: Novelty compared to Boosted Prompt Ensembles (BPE). Earlier methods already build prompt ensembles from difficult examples or reflective failure feedback.

We thank the reviewer for pointing us to BPE. We agree that both methods build an ensemble from the examples the current ensemble gets wrong. The key differences are 1) the optimization objective, and 2) the boosting target (few-shot prompts vs. skills).

Specifically, BPE aims to improve cases where the current ensemble solves in only a few of its samples, so that more prompts vote for the correct answer (improving majority rather than pass@k), whereas our residual-based learning focuses on improving diversity and coverage by targeting the questions that all previous skills fail.

In addition, BPE adds solved training examples as few-shot demonstrations to a prompt. DivSkill-SQL improves the agent's skill, i.e., the instructions that decide how the agent explores the database, uses tools, and checks its answer. This has two practical consequences. First, our skills do not carry examples from the training databases, so they transfer better across dialects and task formats. Second, in the agentic setting, BPE has to resend its demonstrations on every agent turn, which makes it expensive (also see response to Q2 below).

We have added the above detailed discussion to the related work section of the revised paper. To further empirically compare, we adapted BPE to the same agent, tools, and selector, and have the following result:

DialectModelMethodpass@1pass@8Sel.
SQLiteGPT-5.4BPE56.3073.3366.67
SQLiteGPT-5.4DivSkill-SQL62.8784.4471.85
SQLiteOpus-4.6BPE56.2072.5961.48
SQLiteOpus-4.6DivSkill-SQL62.1373.3364.44
BigQueryGPT-5.4BPE47.1469.2757.56
BigQueryGPT-5.4DivSkill-SQL57.8077.5663.41
BigQueryOpus-4.6BPE52.1365.8559.02
BigQueryOpus-4.6DivSkill-SQL62.0773.1764.88

DivSkill-SQL outperforms BPE on both dialects with both model backbones, because of two reasons: 1) BPE sacrifices too much pass@k for improving consistency, and 2) BPE relies on few-shot prompting, which does not help as much as improving skills for Text2SQL agents.

We have added this analysis and results to Section 4.2.

W2 & Q1: The contribution of residual optimization is not clearly ablated (not comparing with prompt optimization techniques like GEPA). How is Opt. Base constructed, and how does it compare with independently optimizing the same eight seeds?

Thank you for raising the question and concern. First we would like to clarify that prompt/skill optimization methods such as GEPA are a component of our framework. We used GEPA as the skill optimizer that improves a single skill on a given residual pool (the SkillOptimizer call in line 8 of Algorithm 1).

We would like to clarify that Opt. Base is the one that runs residual optimization starting from the base skill only, without the eight seed skills. Therefore, to understand what happens if we independently optimize the same eight seeds, we ran and added Ind. Opt. Seed to the Pass@k comparison (Section 4.3, Figure 3) to the revised paper. It runs GEPA on each of the same eight seed skills independently.

In this experiment, DivSkill-SQL has the strongest Pass@k at every k, Ind. Opt. Seed matches Seeds only for k <= 4 and falls to the level of Base for k >= 8. This suggests that, without the residual mechanism forcing them to focus on uncovered cases, independently optimized skills tend to fix the same (easy) failures and lose diversity.

Q2: Actual inference cost and end-to-end latency, including generation, tool use, and selection, and the separate cost of learning the skill bank.

Thank you for the question. We report the explicit cost breakdown below.

Generation and learning. We measure generation cost with Opus-4.6 on BIRD-Critic SQLite. The table below shows the token and time cost for the learning and inference stages.

BPEDivSkill-SQL
Learning
Agent runs16K3.6K
Input tokens1.7B75M
Output tokens8.9M2.9M
Time6 h7.4 h
Inference
Input tokens245K80K
Output tokens1.8K1.8K
Time36 s40 s

At inference, each BPE agent run consumes about 3x more input tokens than a DivSkill-SQL run because it resends its demonstrations on every agent turn (creating very long contexts), at similar latency.

For learning, a residual optimization loop with GPT-5.4 used 3.6K agent runs and 78M tokens in total, including reflection calls (with at most 8 concurrent requests). In comparison, BPE samples ten trajectories for each of 200 training questions in each of its eight rounds, totaling 16K agent runs and 1.7B tokens (over 10x more than DivSkill-SQL).

Selection. Accuracy and cost breakdown with different selectors on Spider2-Lite (GPT-5.4), averaged per question:

AccuracyToken Count and Cost
SelectorSQLiteSnowflakeBigQueryCached InputUncached InputOutputCost ($)Time (s)
Majority vote65.9357.0060.9800000
One-shot71.1158.9462.4404.4K3770.0175.7
Pairwise71.8561.8463.4131.7K27.9K7.5K0.19012.9

Time denotes the per-question latency of selection with at most 8 concurrent requests. Overall, pairwise judge gives highest quality, while one-shot judge could be a reliable lower-cost alternative.

In the revised paper, we have added 1) the detailed generation and learning cost analysis to Section 4.2 ("Cost analysis and comparison with BPE") and 2) selector cost/time results to the selector ablation in Section 4.3.

Q4: How do the learned skills generalize to unseen schemas and query patterns, and what failure modes explain cases where optimization reduces test accuracy compared with the original seeds?

Thank you for the question. We would like to clarify that our main results already evaluate generalization to unseen schemas on a population level. To demonstrate the effect of skill optimization/learning qualitatively, we further included the following simple example in the revised paper:

One example problem we fixed through skill optimization is that the original agent tends to overdo things out of caution: asked for each patient's first lab test, it also listed patients who never had a test. The rule added after optimization was "only keep records that actually have a matching entry when the question requires one." In our runs, this preference fixes 6 cases.

However, this might also lead to a failure mode when the wording is loose. In one debugging task, the user asked to fix a query that returns four columns and mentioned wanting "the complete test result data." The optimized skill took this literally and returned all 44 columns of the table. The original skill kept the user's four columns, which was the expected fix.

We sincerely appreciate your time and valuable feedback, which has helped us improve our work.

With the discussion period ending in a day, we look forward to your thoughts on our response. Please let us know if there is anything more we can do to address your comments!

Thanks again for your time and feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.