- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper concerns text-to-SQL modeling in the face of ambiguous or incomplete user inputs. The authors propose a pipeline method with 3 stages, using an LLM to ask clarifying questions, generate a pool of candidate SQL queries, execute those queries and get return values, and compare across candidates and ask more follow up questions. They use an LLM user simulator for the conversational sim. Every interaction has a fixed budget of follow up questions it is allowed. The authors also ran evals varying the interaction budget and the candidate pool size and provided commentary on the impact of these variables. Using the pipeline method, the authors claimed +5–12 pp over the best baseline on three text-to-SQL benchmarks.
Critical strengths
- Clarity: paper is well written and clear
- Well scoped: paper targets a specific problem in a niche domain and avoids being overly broad
- Clear and reproducible protocol: baselines reimplemented, tasks and budgets consistent
Critical weaknesses
-
The paper claims gains over a full information baseline (Figures 3 and 7) but no explanation for this is provided or analysis done. How can asking for more information outperform a model with all the pertinent information provided up front? What information was gained (e.g. Figure 3, between P=3->11 for D=16)? Was that information missing in the full-information baseline? Was there any oracle leakage from the sim?
-
The full method (DuetSQL) is not demonstrated to have statistically significant performance over the bare loop. The two are never compared directly, and none of the experiments have errors bars or confidence intervals so the reader cannot tell if any of the results are statistically significant. DuetSQL beats the bare loop by only 2.2, 3.3 and 4.5 pp (P = 0, 3, 7), against a binomial 95% CI on the difference of roughly ±5.5 pp at n = 600. The added stages, execution and early stopping are not shown to contribute.
-
No comparison to a simpler alternative. There’s no baseline of a reasoning model using SQL as a tool (AI industry standard); reasoning effort is never varied; and the solver is flash-tier despite the authors having access to a stronger pro model for the simulator.
-
Novelty is limited. The method is a pipeline of standard methods + prompting, and is incremental over ClarifyGPT. Combined with #3 above severely limits novelty.
-
Cost is high and partly unreported. The number of separate model calls is not reported, which by my estimate could be upwards of 60 calls per task at P=9, about 21× the tokens of Conv. Latency isn’t measured.
Questions for authors
biggest questions are #1 and #3 above
12 comments
Question 2: What does complete DuetSQL gain over the bare loop, and are the gains statistically significant?
Question 2: What does complete DuetSQL gain over the bare loop, and are the gains statistically significant?
-
Higher success with lower resource use.
We now directly compare complete DuetSQL with the Section 3 bare loop using the same solver and question allowance. At our main budgets of interest, P=5 and P=7, DuetSQL achieves higher success with fewer solver tokens and questions:
P Bare loop → DuetSQL success (%) Gain and 95% CI (pp) Pointwise p Solver-token reduction Question reduction 5 38.00 → 42.33 +4.33 [1.00, 7.67] 0.0167 44.7% 13.9% 7 39.33 → 43.83 +4.50 [1.00, 8.00] 0.0150 39.0% 18.4% This joint improvement is the practical benefit of the complete pipeline: the additional stages help use interaction and computation more efficiently.
-
Statistical significance.
Because both methods are evaluated on the same tasks, we use paired bootstrap confidence intervals and two-sided exact McNemar tests. The gains at P=5 and P=7 are significant in pointwise tests, and both 95% confidence intervals exclude zero. After Holm correction across all four examined budgets (P=0/3/5/7), however, both adjusted p-values are 0.0599. The gains therefore do not remain significant at the 0.05 level after correction; our statistical claim is limited to the pointwise comparisons.
The component ablations clarify the distinct roles of the additional stages:
-
Request Clarification primarily targets computational efficiency. It obtains requirements before candidate sampling, so initial questions do not each require a SQL pool. The paired analysis shows 31.3% fewer solver tokens (95% CI [29.0%, 33.5%]), supporting this cost objective. Its observed mean success gain of 1.13 percentage points is not statistically significant (Holm-adjusted p=0.767). Thus, the evidence supports its intended contribution: reducing the computation needed for early clarification.
-
Output Specification improves task success by checking shared omissions. Candidates can agree while jointly omitting a required output item or ordering. This stage explicitly confirms these requirements with the user before retaining or revising the SQL. Its mean success gain of 4.60 percentage points is statistically significant (95% CI [3.07, 6.20]; Holm-adjusted p=1.65×10⁻⁸), supporting the value of this final requirement check.
-
Early stopping contributes resource control. It ends further sampling and questioning once candidate executions sufficiently agree. At P=9, it saves 25.4% of solver tokens and 22.0% of questions, with respective 95% CIs of [22.4%, 28.4%] and [19.7%, 24.4%]. The observed success difference of −0.83 percentage points is not statistically significant (95% CI [−2.83, 1.17]; p=0.522). These results support its resource-control role, while accuracy equivalence remains unestablished.
These comparisons assess whole-stage effects; the independent contribution of diagnostic execution remains unisolated.
Question 3: Stronger solvers, SQL-tool agents, and reasoning effort
Question 3: Stronger solvers, SQL-tool agents, and reasoning effort
-
Research scope and model settings.
Our study starts from the observation that allowing more user questions does not consistently improve SQL accuracy. We therefore ask how the evidence used to formulate clarification questions affects the benefit of additional interaction. Performance and the magnitude of gains can vary across models and inference settings, so we hold the solver configuration fixed when comparing clarification strategies and interaction budgets. The paper's Qwen evaluation separately examines these findings with a different solver.
We agree that stronger solvers and reasoning effort are also important to evaluate. Following your suggestion, we add two studies with DeepSeek-Pro-0813: a comparison with an autonomous SQL-tool agent, and a sensitivity analysis of success and cost across reasoning-effort settings at a fixed interaction budget.
-
Comparison with a Pro SQL-tool agent.
BIRD-Interact distinguishes conversational clarification under a predefined protocol (c-Interact) from autonomous interaction with the user and database (a-Interact). Our original study focuses on conversational clarification, which motivated the selection of baselines representing different clarification strategies. An autonomous SQL-tool agent additionally controls execution, revision, and when to ask questions, making it a broader system-level comparison. We add this comparison to test whether DuetSQL remains useful when the alternative also uses a Pro reasoning solver with SQL tools.
Both systems use Pro/high, the same 600 tasks and public inputs, the same user simulator, and P=5. The tool agent autonomously asks questions, executes and revises SQL, and submits its answer. SQL execution and model-turn counts are uncapped. Both systems use the same terminal execution-error handling, without ground-truth correctness feedback.
Method Correct / 600 Success rate Autonomous SQL-tool agent 127/600 21.17% DuetSQL 236/600 39.33% DuetSQL improves success by 18.17 percentage points (paired bootstrap 95% CI [14.50, 22.00]; two-sided exact McNemar p=2.17×10⁻²⁰), with all failures retained in the denominator. This shows that the advantage is also observed with Pro as the solver. The comparison matches the solver and question allowance, but not total computation; the result concerns this tool-agent implementation and setting.
-
Effect of reasoning effort at a fixed interaction budget.
Internal reasoning computation is a separate resource dimension from user interaction. Following your suggestion, we vary Pro-0813's reasoning effort within DuetSQL on the same subset of the 600 tasks, holding the interaction allowance fixed at P=5.
Reasoning effort Correct / 120 Success rate Mean solver tokens/task (M) low 51/120 42.50% 0.635 high 53/120 44.17% 0.745 max 54/120 45.00% 0.816 Token averages use matched complete records.
These results describe how observed success and token use vary across reasoning-effort settings at a fixed interaction allowance, complementing the paper's Qwen evaluation of a different solver.
Preface and Question 1: Why can interaction outperform full-information, and is there simulator leakage?
Preface
Thank you for the careful review, particularly the questions about the full-information reference and stronger reasoning agents. Our starting point is a practical difficulty in interactive Text-to-SQL: user requests can leave important requirements unspecified, yet giving an agent more opportunities to ask questions does not reliably improve its final SQL. We study what makes additional interaction useful, focusing on the evidence that guides clarification and how it changes the gains from a larger question budget.
This motivates two linked evaluations. First, we examine how clarification evidence affects interaction-budget scaling, comparing representative methods and questioning from context alone, a single SQL, and SQL contrasts under a shared protocol. The solver and simulator configurations are held fixed within each comparison so that changes in model choice do not drive the budget curves. The accompanying dialogue analyses also address why clarification can help and why interaction can sometimes outperform the full-information reference. Second, we evaluate how these findings can guide a practical system that balances task success with interaction and computation costs. The goal is to identify useful cost–benefit trade-offs under resource constraints, rather than pursue the highest benchmark accuracy alone. The additional analyses below address your concerns about information provenance, statistical uncertainty, autonomous tools, reasoning effort, and deployment costs.
Question 1: Why can interaction outperform full-information, and is there simulator leakage?
What “full-information” provides, and why clarification can still help.
BIRD-Interact's original task annotations provide the complete user request, unmasked background knowledge (e.g., business terms and formulas), and schema and column descriptions. In the full-information setting, the same solver receives all of this annotated information upfront and must still translate it into SQL; the reference SQL is not provided.
Users usually describe the business result they want, without anticipating the questions about counting or combining records that arise during SQL implementation. Yet they may have a clear sense of what that result should mean. If the agent explains how alternative implementations would change the result, the user can judge which one matches their intent without knowing SQL. Clarification can therefore elicit intent that was present but unstated in the original request, giving the solver guidance beyond the static annotations.
Two examples of unstated requirements.
museum_artifact_16, Section 3.3: the interactive SQL uses a medium-risk default specified through clarification; the full-information SQL excludes missing values from the average.)reverse_logistics_3: the solver assumes a ratio of total recovered value to total elapsed days; clarification establishes that the intended calculation averages the per-return daily recovery rates.)In both scenarios, the annotations supply the formula, while clarification establishes how the user wants it applied.
Was there oracle leakage from the simulator?
We examine this concern through human review of answers and analysis of question forms, using the same 129 reviewed question-answer turns from 10 tasks across DuetSQL, ClarifyGPT, and Conv.
First, a single annotator reviewed 129 turns: 122/129 (94.6%) were labeled as having no observed leakage, 4 as suspected leakage, and 3 as uncertain.
Second, a preliminary model-assisted classification found explicit alternatives in 94/129 questions (72.9%). For example, the agent asks whether a failure timestamp refers to record creation or match occurrence, and the simulator selects record creation. Here, the agent has identified the ambiguity before asking the user to confirm their intent; such confirmation need not supply a SQL solution. This percentage measures question form, not whether answers merely select an option or are leakage-free.
These observations are consistent with clarification of user intent, although the sample audit cannot fully rule out reference-SQL influence.