- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose an ensemble method for text-to-sql generation, where prompts are constructed such that it addresses the residual failures of the ensemble. By constructing their prompts this way, they are able to build a diverse set of prompts suitable for ensemble. During inference, they apply an LLM judge to select the right prompt for the right problem.
Critical strengths
- The authors achieve significant improvements over other methods.
- The authors propose an intuitive method to for text-to-sql ensembles.
Critical weaknesses
- The proposed method is compute intensive. In addition to running multiple prompts, the authors require additional pairwise LLM judges during inference, making it very costly.
- Experiments. The authors do not analyse the effect of number of ensemble components.
Questions for authors
How much time does the LLM judges incur?
2 comments
We sincerely appreciate your time and valuable feedback, which has helped us improve our work.
With the discussion period ending in a day, we look forward to your thoughts on our response. Please let us know if there is anything more we can do to address your comments!
Thanks again for your time and feedback!
We thank the reviewer for the helpful feedback, and address each of the concerns below. All changes in the revised paper are highlighted in blue for convenience of reading (we will change it back after the rebuttal ends).
W1 & Q1: The method is compute-intensive. It runs multiple prompts and adds pairwise LLM judges at inference. How much time do the LLM judges incur?
We thank the reviewer for raising this concern and bringing in this dimension. We agree that pairwise judging grows quadratically with the number of candidates and can become expensive at large K. However, in practice DivSkill-SQL does not need to scale beyond K = 8, and a much smaller ensemble is often enough: as we later show in W2, with only K = 2 or K = 4 candidates DivSkill-SQL already reaches decent performance. Moreover, the pairwise comparisons for the same question share a large amount of context, which is served as cached input. As shown in the cost breakdown below, more than half of the pairwise judge's input tokens are cached, which costs 90% less than standard (uncached) input tokens. This substantially reduces the actual cost of pairwise judging.
We would also like to clarify that we adopt the pairwise judge from CHASE-SQL to reach higher final quality, and this is a replaceable component of our system. To better understand quality-cost trade-off of the judge, we additionally compare with a one-shot judge that compares all distinct candidates in a single LLM call, and majority voting. The pairwise judge leverages parallelism due to independence between comparisons, and it runs with at most 8 concurrent processes (aligned with learning stage). We report the accuracy, token cost breakdown and time below:
The results indicate that one-shot judge could be a decent low-budget alternative. Importantly, even with this cheap judge, DivSkill-SQL still outperforms CHASE-SQL with the pairwise judge on all three dialects. Users can therefore choose the selector according to their budget.
We have added the above results and analysis to the revised paper (Section 4.3, "Choice of selector").
W2: The effect of the number of ensemble components is not analyzed.
We thank the reviewer for the suggestion, and conducted an ablation on the number of candidates k:
DivSkill-SQL achieves higher Pass@k and selected accuracy than CHASE-SQL for every k, and with only two candidates it already outperforms CHASE-SQL with eight in selected accuracy.
We have added the results and analysis to the revised paper (Section 4.3, "Number of ensemble components").