acceptodds
Under review as a conference paper at ICLR 2027

What Drives Interaction Budget Scaling in Text-to-SQL?

Abstract

Interactive Text-to-SQL can resolve underspecified requirements through user clarification, yet larger interaction budgets do not consistently yield better SQL solutions. We study what makes additional interaction useful by comparing five representative methods and conducting controlled experiments on clarification evidence in Bird-Interact-Full. Context-only questioning is competitive initially, while guiding questions with differences between as few as two SQL candidates yields larger gains as the interaction budget increases. Wider pools bring earlier gains, but their advantage narrows with further interaction. Building on these findings, we introduce , which combines context-guided requirement elicitation with iterative SQL-contrast-guided clarification and uses execution agreement to control further sampling and questioning. Across three benchmarks, DuetSQL outperforms the strongest compared methods by 5-12 percentage points at matched interaction allowances. DuetSQL configurations also lie on the empirical Pareto frontiers in our comparisons under shared resource limits, offering favorable trade-offs between task success and resource use.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Text-to-SQL, Interactive Agents, User Clarification, Interaction Budget Scaling, Cost-Aware Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zQi9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper concerns text-to-SQL modeling in the face of ambiguous or incomplete user inputs. The authors propose a pipeline method with 3 stages, using an LLM to ask clarifying questions, generate a pool of candidate SQL queries, execute those queries and get return values, and compare across candidates and ask more follow up questions. They use an LLM user simulator for the conversational sim. Every interaction has a fixed budget of follow up questions it is allowed. The authors also ran evals varying the interaction budget and the candidate pool size and provided commentary on the impact of these variables. Using the pipeline method, the authors claimed +5–12 pp over the best baseline on three text-to-SQL benchmarks.

Critical strengths
  1. Clarity: paper is well written and clear
  2. Well scoped: paper targets a specific problem in a niche domain and avoids being overly broad
  3. Clear and reproducible protocol: baselines reimplemented, tasks and budgets consistent
Critical weaknesses
  1. The paper claims gains over a full information baseline (Figures 3 and 7) but no explanation for this is provided or analysis done. How can asking for more information outperform a model with all the pertinent information provided up front? What information was gained (e.g. Figure 3, between P=3->11 for D=16)? Was that information missing in the full-information baseline? Was there any oracle leakage from the sim?

  2. The full method (DuetSQL) is not demonstrated to have statistically significant performance over the bare loop. The two are never compared directly, and none of the experiments have errors bars or confidence intervals so the reader cannot tell if any of the results are statistically significant. DuetSQL beats the bare loop by only 2.2, 3.3 and 4.5 pp (P = 0, 3, 7), against a binomial 95% CI on the difference of roughly ±5.5 pp at n = 600. The added stages, execution and early stopping are not shown to contribute.

  3. No comparison to a simpler alternative. There’s no baseline of a reasoning model using SQL as a tool (AI industry standard); reasoning effort is never varied; and the solver is flash-tier despite the authors having access to a stronger pro model for the simulator.

  4. Novelty is limited. The method is a pipeline of standard methods + prompting, and is incremental over ClarifyGPT. Combined with #3 above severely limits novelty.

  5. Cost is high and partly unreported. The number of separate model calls is not reported, which by my estimate could be upwards of 60 calls per task at P=9, about 21× the tokens of Conv. Latency isn’t measured.

Questions for authors

biggest questions are #1 and #3 above

12 comments
AuthorsOpenReview ↗

Preface and Question 1: Why can interaction outperform full-information, and is there simulator leakage?

Preface

Thank you for the careful review, particularly the questions about the full-information reference and stronger reasoning agents. Our starting point is a practical difficulty in interactive Text-to-SQL: user requests can leave important requirements unspecified, yet giving an agent more opportunities to ask questions does not reliably improve its final SQL. We study what makes additional interaction useful, focusing on the evidence that guides clarification and how it changes the gains from a larger question budget.

This motivates two linked evaluations. First, we examine how clarification evidence affects interaction-budget scaling, comparing representative methods and questioning from context alone, a single SQL, and SQL contrasts under a shared protocol. The solver and simulator configurations are held fixed within each comparison so that changes in model choice do not drive the budget curves. The accompanying dialogue analyses also address why clarification can help and why interaction can sometimes outperform the full-information reference. Second, we evaluate how these findings can guide a practical system that balances task success with interaction and computation costs. The goal is to identify useful cost–benefit trade-offs under resource constraints, rather than pursue the highest benchmark accuracy alone. The additional analyses below address your concerns about information provenance, statistical uncertainty, autonomous tools, reasoning effort, and deployment costs.

Question 1: Why can interaction outperform full-information, and is there simulator leakage?

  1. What “full-information” provides, and why clarification can still help.

    BIRD-Interact's original task annotations provide the complete user request, unmasked background knowledge (e.g., business terms and formulas), and schema and column descriptions. In the full-information setting, the same solver receives all of this annotated information upfront and must still translate it into SQL; the reference SQL is not provided.

    Users usually describe the business result they want, without anticipating the questions about counting or combining records that arise during SQL implementation. Yet they may have a clear sense of what that result should mean. If the agent explains how alternative implementations would change the result, the user can judge which one matches their intent without knowing SQL. Clarification can therefore elicit intent that was present but unstated in the original request, giving the solver guidance beyond the static annotations.

  2. Two examples of unstated requirements.

    • Missing-value handling in an aggregate. The formula is provided, but it does not state how missing inputs should contribute. (museum_artifact_16, Section 3.3: the interactive SQL uses a medium-risk default specified through clarification; the full-information SQL excludes missing values from the average.)
    • Aggregation of individual rates. The annotations give the per-return formula but leave the overall aggregation rule unstated. (reverse_logistics_3: the solver assumes a ratio of total recovered value to total elapsed days; clarification establishes that the intended calculation averages the per-return daily recovery rates.)

    In both scenarios, the annotations supply the formula, while clarification establishes how the user wants it applied.

  3. Was there oracle leakage from the simulator?

    We examine this concern through human review of answers and analysis of question forms, using the same 129 reviewed question-answer turns from 10 tasks across DuetSQL, ClarifyGPT, and Conv.

    First, a single annotator reviewed 129 turns: 122/129 (94.6%) were labeled as having no observed leakage, 4 as suspected leakage, and 3 as uncertain.

    Second, a preliminary model-assisted classification found explicit alternatives in 94/129 questions (72.9%). For example, the agent asks whether a failure timestamp refers to record creation or match occurrence, and the simulator selects record creation. Here, the agent has identified the ambiguity before asking the user to confirm their intent; such confirmation need not supply a SQL solution. This percentage measures question form, not whether answers merely select an option or are leakage-free.

    These observations are consistent with clarification of user intent, although the sample audit cannot fully rule out reference-SQL influence.

AuthorsOpenReview ↗

Question 2: What does complete DuetSQL gain over the bare loop, and are the gains statistically significant?

Question 2: What does complete DuetSQL gain over the bare loop, and are the gains statistically significant?

  1. Higher success with lower resource use.

    We now directly compare complete DuetSQL with the Section 3 bare loop using the same solver and question allowance. At our main budgets of interest, P=5 and P=7, DuetSQL achieves higher success with fewer solver tokens and questions:

    PBare loop → DuetSQL success (%)Gain and 95% CI (pp)Pointwise pSolver-token reductionQuestion reduction
    538.00 → 42.33+4.33 [1.00, 7.67]0.016744.7%13.9%
    739.33 → 43.83+4.50 [1.00, 8.00]0.015039.0%18.4%

    This joint improvement is the practical benefit of the complete pipeline: the additional stages help use interaction and computation more efficiently.

  2. Statistical significance.

    Because both methods are evaluated on the same tasks, we use paired bootstrap confidence intervals and two-sided exact McNemar tests. The gains at P=5 and P=7 are significant in pointwise tests, and both 95% confidence intervals exclude zero. After Holm correction across all four examined budgets (P=0/3/5/7), however, both adjusted p-values are 0.0599. The gains therefore do not remain significant at the 0.05 level after correction; our statistical claim is limited to the pointwise comparisons.

The component ablations clarify the distinct roles of the additional stages:

  • Request Clarification primarily targets computational efficiency. It obtains requirements before candidate sampling, so initial questions do not each require a SQL pool. The paired analysis shows 31.3% fewer solver tokens (95% CI [29.0%, 33.5%]), supporting this cost objective. Its observed mean success gain of 1.13 percentage points is not statistically significant (Holm-adjusted p=0.767). Thus, the evidence supports its intended contribution: reducing the computation needed for early clarification.

  • Output Specification improves task success by checking shared omissions. Candidates can agree while jointly omitting a required output item or ordering. This stage explicitly confirms these requirements with the user before retaining or revising the SQL. Its mean success gain of 4.60 percentage points is statistically significant (95% CI [3.07, 6.20]; Holm-adjusted p=1.65×10⁻⁸), supporting the value of this final requirement check.

  • Early stopping contributes resource control. It ends further sampling and questioning once candidate executions sufficiently agree. At P=9, it saves 25.4% of solver tokens and 22.0% of questions, with respective 95% CIs of [22.4%, 28.4%] and [19.7%, 24.4%]. The observed success difference of −0.83 percentage points is not statistically significant (95% CI [−2.83, 1.17]; p=0.522). These results support its resource-control role, while accuracy equivalence remains unestablished.

These comparisons assess whole-stage effects; the independent contribution of diagnostic execution remains unisolated.

AuthorsOpenReview ↗

Question 3: Stronger solvers, SQL-tool agents, and reasoning effort

Question 3: Stronger solvers, SQL-tool agents, and reasoning effort

  1. Research scope and model settings.

    Our study starts from the observation that allowing more user questions does not consistently improve SQL accuracy. We therefore ask how the evidence used to formulate clarification questions affects the benefit of additional interaction. Performance and the magnitude of gains can vary across models and inference settings, so we hold the solver configuration fixed when comparing clarification strategies and interaction budgets. The paper's Qwen evaluation separately examines these findings with a different solver.

    We agree that stronger solvers and reasoning effort are also important to evaluate. Following your suggestion, we add two studies with DeepSeek-Pro-0813: a comparison with an autonomous SQL-tool agent, and a sensitivity analysis of success and cost across reasoning-effort settings at a fixed interaction budget.

  2. Comparison with a Pro SQL-tool agent.

    BIRD-Interact distinguishes conversational clarification under a predefined protocol (c-Interact) from autonomous interaction with the user and database (a-Interact). Our original study focuses on conversational clarification, which motivated the selection of baselines representing different clarification strategies. An autonomous SQL-tool agent additionally controls execution, revision, and when to ask questions, making it a broader system-level comparison. We add this comparison to test whether DuetSQL remains useful when the alternative also uses a Pro reasoning solver with SQL tools.

    Both systems use Pro/high, the same 600 tasks and public inputs, the same user simulator, and P=5. The tool agent autonomously asks questions, executes and revises SQL, and submits its answer. SQL execution and model-turn counts are uncapped. Both systems use the same terminal execution-error handling, without ground-truth correctness feedback.

    MethodCorrect / 600Success rate
    Autonomous SQL-tool agent127/60021.17%
    DuetSQL236/60039.33%

    DuetSQL improves success by 18.17 percentage points (paired bootstrap 95% CI [14.50, 22.00]; two-sided exact McNemar p=2.17×10⁻²⁰), with all failures retained in the denominator. This shows that the advantage is also observed with Pro as the solver. The comparison matches the solver and question allowance, but not total computation; the result concerns this tool-agent implementation and setting.

  3. Effect of reasoning effort at a fixed interaction budget.

    Internal reasoning computation is a separate resource dimension from user interaction. Following your suggestion, we vary Pro-0813's reasoning effort within DuetSQL on the same subset of the 600 tasks, holding the interaction allowance fixed at P=5.

    Reasoning effortCorrect / 120Success rateMean solver tokens/task (M)
    low51/12042.50%0.635
    high53/12044.17%0.745
    max54/12045.00%0.816

    Token averages use matched complete records.

    These results describe how observed success and token use vary across reasoning-effort settings at a fixed interaction allowance, complementing the paper's Qwen evaluation of a different solver.

Reviewer_QuJdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper explores why allowing AI systems to ask more clarification questions does not always improve performance in interactive Text-to-SQL tasks. Authors tested several methods and found that questions based on differences between multiple possible SQL queries are more effective than questions based only on context or a single SQL query. Authors then create DuetSql to combat this. Duetsql does 3 things: 1)Asks questions to better understand the user's request, 2) Generates multiple SQL queries, compares their differences, and asks targeted questions to resolve uncertainty, 3) Confirms that the output matches the user's requirements. They then use 3 different benchmarks to show increase in performance by 5-12% compared to other methods.

Critical strengths
  1. Substantial empirical gains with transparent token usage is shown. Paper reports everything from questions to token count alongside many different ablations. Kudos to making this transparent.

  2. Authors do a novel ablation by comparing and contrasting at least two candidate SQLs. This ablation demonstrates where the system’s non-trivial gains come from.

Critical weaknesses

Deepseek is the only simulated user. There are 2 issues here. First, this is an AI simulation and not real users. There should be some human robustness check to show performance on real end users. Second concern is that only Deepseek was chosen as the simulator. While authors do state that this is an open-source model, at least 1-2 other model  families should be used to simulate the end users, and for cost, perhaps a subset of analysis can be run on more expensive, proprietary models. The authors should consider adding BOTH, as it would show their method is more generalizable.

Questions for authors
  1. You describe 14–16 % of tasks still failing at P = 9. It would be good to address aside from performance, what caused these tasks to fail.

  2. Have you conducted pilot tests with human users to see if the style of an AI user is synonymous to a real user?

1 comment

Human validation and simulator behavior

Thank you for your careful review and constructive comments. We have not yet conducted a live-user pilot or directly compared simulator and human response styles. Our study focuses on clarification strategies and interaction-budget scaling within a simulated-user benchmark framework, which supports comparisons under a shared user protocol. How well this framework represents real-user behavior is an important question of benchmark validity that we have not directly investigated. We will make this scope explicit and identify human validation as follow-up work.

As a limited check of simulator responses, a single annotator reviewed 129 question-answer turns across DuetSQL, ClarifyGPT, and Conv: 122/129 (94.6%) were labeled as having no observed leakage, with 4 suspected and 3 uncertain. This content audit examines potential leakage; it does not establish similarity to human response styles or robustness to real users.

Robustness across user-simulator families

Our main comparisons use a shared user-simulator configuration across methods to study clarification strategies under a common interaction protocol. Following your suggestion, we additionally evaluated Qwen 2.4T as the user simulator, which meets UserSim-Guard's reliability thresholds in our evaluation. We compared all three methods on the same 600 BIRD-Interact-Full tasks, holding the DeepSeek-v4.1-Flash solver and P=5 fixed.

User simulatorDuetSQLClarifyGPTConv
DeepSeek-v4-Pro40.33%33.17%33.67%
Qwen 2.4T40.67%34.50%33.00%

With the Qwen simulator, DuetSQL achieves 40.67% success and continues to lead ClarifyGPT and Conv by 6.17 and 7.67 percentage points, respectively. These results show that DuetSQL's observed advantage extends to another simulator family at the tested P=5 budget.

Remaining failures at high interaction budgets

We analyzed the remaining failures on BIRD-Interact-Full at P=9. Of 326 failed tasks, 259 (79.4%) produced executable SQL but failed task evaluation, showing that successful execution alone is insufficient to establish correctness.

We inspected 20 randomly sampled cases from these 259 failures. The main observed issues were:

  • Unconfirmed or omitted requirements (7/20; 35%). The solver made unsupported assumptions about unresolved details or omitted necessary requirements.
  • Clarified requirements not implemented (2/20; 10%). Explicit answers were not reflected in the final SQL. For example, a confirmed sorting requirement was omitted in solar_panel_13.
  • Other SQL construction errors (2/20; 10%). The SQL contained incorrect field mappings or numerical operations.
  • Mixed discrepancies (8/20; 40%). These involved conflicts between simulator answers and fixed references/tests, numerical types or precision, unspecified output names, or an extra constraint introduced by our SQL. Their contributions to failure are not fully isolated, so this group should not be interpreted as a benchmark-error rate.

One further case (1/20; 5%) fell outside these groups. Across the categories above, 4/20 cases (20%) triggered consistency-based stopping even though the selected SQL failed evaluation. This overlapping observation shows that candidate agreement can miss errors and does not itself establish correctness.

These sample-level findings suggest that a larger question allowance alone cannot ensure that the right requirements are elicited and implemented. They motivate checking clarification coverage, verifying that confirmed answers are reflected in SQL, and validating correctness independently of candidate agreement, alongside checking alignment between simulator feedback and evaluation.

Reviewer_Et78OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper finds that five existing interactive Text-to-SQL methods show next to no gains with higher conversation budgets, and controlled experiments show that questions guided by contrasts between SQL candidates scale better than context-only or single-SQL questions. Building on this, the proposed DuetSQL combines context-guided and SQL-contrast-guided clarification, outperforming the strongest baseline by 5–12 points across three benchmarks at favorable resource trade-offs.

Critical strengths

S1 The paper addresses a well-motivated question that is often overlooked: whether more interaction turns actually help for text-to-SQL. Prior work implicitly assumes that a larger interaction budget translates into better results, and the external-method analysis shows clearly that it often does not.

S2 The controlled study yields clear, actionable findings.

S3 The analysis directly motivates the method design. Each DuetSQL stage maps to a finding, which makes the approach easy to follow, and the method yields notable gains.

S4 The resource analysis is fairly thorough: tokens and questions are reported per budget, Pareto plots compare methods under a shared resource cap, and the ablations include cost.

Critical weaknesses

W1 Missing grounding in human user interaction. The paper acknowledges that “further questions demand user effort” and treats the question count as a cost. However, it does not cite or discuss user research on clarification fatigue or how tolerant users are of many questions.

W2 The evaluation is done with simulated users only. The simulator was validated on USERSIM-GUARD, but not on the kinds of questions DuetSQL actually asks. Contrast-guided questions that expose concrete SQL differences may be more likely to elicit solution-revealing answers. This could also partly explain why the method surpasses the full-information reference. A small human study, or a leakage audit on DuetSQL’s own dialogues, would strengthen the claims.

W3 Token counts are reported and DuetSQL compares favorably with ATD and PleaSQLarify and is very cheap on AMBROSIA-Interact. Still, it uses about 13× the tokens of BIRD-Interact Conv and more than ClarifyGPT. A short discussion of deployability, such as cost per task or latency per turn compared with the baselines, would help.

W4 All results come from a single sampled trajectory with no repeated seeds and no confidence intervals. With 600 tasks, the standard error is roughly 2 points, so several claimed differences fall within noise.

Questions for authors

Q1 Did the authors check DuetSQL’s dialogues for the simulator revealing solution details, given that contrast-guided questions show concrete SQL differences? What explains contrast-based clarification exceeding the full-information reference?

Q2 Can the authors report confidence intervals or paired significance tests for the key comparisons?

Q3 The reproducibility statement does not mention whether AMBROSIA-Interact will be released publicly. Do the authors plan to release the benchmark so others can evaluate on it? It would be a useful contribution to the field.

Writing suggestion:

  • Section 3.2 is unnecessarily number-heavy. Most of the values can be read from Figure 3 and Table 6. Rewriting the section around the narrative and core findings, with only one or two anchor numbers, would improve readability.
2 comments
AuthorsOpenReview ↗

User effort and clarification fatigue & Simulator leakage and the full-information reference

User effort and clarification fatigue

Thank you for highlighting this gap. We will better ground our user-cost measure in human interaction research.

  1. Why we count questions.

    Our study examines how task success changes with additional user interaction. Question count tracks how often a method requests user input, providing a reproducible measure of interaction demand under a shared protocol. We report it alongside task success and solver computation to characterize their trade-offs.

  2. Evidence and limitations.

    In product search, Zou et al. (2020) found that about half of participants reported willingness to answer 6–10 clarification questions, while some stopped due to fatigue. Zou et al. (2023) further showed that question quality and ordering affect user behavior and satisfaction. These findings motivate treating question count as a coarse proxy for user effort: it does not capture comprehension and answering difficulty, multiple subquestions within a turn, waiting time, or accumulated fatigue. Tolerance observed in search tasks cannot establish a general threshold for SQL clarification.

  3. Implications for our evaluation.

    Our budget curves describe the trade-off between task success and interaction demand. These curves do not establish human tolerance limits. We will add the literature and clarify these boundaries in the manuscript. Direct human validation remains future work, including answering time, perceived burden, and reasons for stopping.

Simulator leakage and the full-information reference

We agree that general simulator validation should be complemented by checks on the questions DuetSQL actually asks.

  1. Human review of actual dialogues.

    A single annotator reviewed 129 question-answer turns from 10 tasks across DuetSQL, ClarifyGPT, and Conv. 122/129 (94.6%) were labeled as having no observed leakage, with 4 suspected and 3 uncertain. The review includes 13 turns from DuetSQL's contrast-guided loop, all labeled as having no observed leakage.

  2. The agent often identifies the alternatives before asking.

    Across the same reviewed sample, a preliminary model-assisted classification found that 94/129 questions (72.9%) offered two or more explicit alternatives. The agent has already articulated the alternatives, so the user can clarify intent by choosing between them. For example, DuetSQL asks whether titles tied on total content should be ordered by identifier or may appear in any order; the simulator confirms that any order is acceptable. This illustrates confirmation of an intended result without supplying SQL. The percentage describes question form, not a non-leakage rate.

  3. Why clarification can help beyond full-information.

    The full-information reference supplies the dataset's annotated request, background knowledge, and schema information, but not the reference SQL. Users may leave data-handling choices unstated when describing a task, yet can judge which outcome matches their intent when asked. Section 3.3 illustrates this with missing-value handling: the formula is provided, while clarification specifies a medium-risk default for missing inputs to the average. This adds a convention left unstated in the original annotations.

    These observations are consistent with clarification of user intent, although the small audit cannot fully rule out reference-SQL influence.

AuthorsOpenReview ↗

Deployment cost and response time & Repeatability and paired statistical evidence & Public release of AMBROSIA-Interact

Deployment cost and response time

Thank you for this suggestion. In the paper's BIRD-Interact-Full runs, DuetSQL uses approximately 13 times Conv's tokens at P=5. We assess this additional computation alongside the improvement in task success.

The table reports reference API costs from recorded solver tokens, including repairs and excluding user simulation, evaluation, and database/hosting costs. Resource means use complete logged trajectories.

MethodPSuccess rate (%)Solver cost (US cents/task)Estimated mean RT (s/turn)
Conv327.330.8410.66
Conv527.170.8610.63
ClarifyGPT331.506.9325.12
ClarifyGPT531.336.9324.82
DuetSQL340.507.0422.05
DuetSQL542.3311.3927.55
  • At P=3, DuetSQL improves success over ClarifyGPT by 9.00 percentage points (40.50% vs 31.50%) at nearly the same cost: 7.04 vs 6.93 US cents per task.
  • At P=5, the success advantage is 11.00 percentage points (42.33% vs 31.33%), with higher cost: 11.39 vs 6.93 US cents per task.

Conv remains the least expensive option. DuetSQL offers higher observed task success at additional cost, with P=3 providing a lower-cost choice than P=5. The appropriate method and budget depend on the application's accuracy and resource requirements.

Latency estimates. Mean RT is a scheduling projection from recorded call durations, estimating model-processing time per response under concurrency and question pre-generation. It excludes database execution and other system overhead and is not measured end-to-end latency. Success and costs describe the original runs; the projected scheduling variant has not been re-evaluated for accuracy or token use. We will include these costs and clearly identified latency estimates in the revised resource discussion.

Repeatability and paired statistical evidence

We report complementary evidence from repeated runs and paired analysis in the supplementary v4.1-Flash evaluation on BIRD-Interact-Full at P=5.

  1. Results across independent runs.

    We added two independent executions of the complete DuetSQL pipeline. Across three runs on the same 600 tasks, success rates were 40.33%, 36.83%, and 39.33%, giving a mean of 38.83% and a sample standard deviation of 1.80 percentage points. All three exceeded the existing ClarifyGPT run's success rate of 33.17%, so the observed advantage was not confined to a single DuetSQL execution.

  2. Paired comparison with ClarifyGPT.

    Comparing the first DuetSQL run with the existing ClarifyGPT run gives 40.33% versus 33.17% success: a gain of 7.17 percentage points, with a paired task-bootstrap 95% CI of [3.67, 10.67] and a two-sided exact McNemar p-value of 0.000115. This provides statistical support for DuetSQL's advantage in this matched comparison.

The standard deviation describes variation across runs; the paired confidence interval describes task-level uncertainty conditional on the first pair of runs.

Public release of AMBROSIA-Interact

Yes. We are committed to publicly releasing AMBROSIA-Interact. The temporary anonymous review release provides the exact 300-task evaluation set used in our experiments, its associated databases, preprocessing and evaluation code, and usage instructions. The interaction protocol and user-simulator prompt are also included. These materials allow researchers to reconstruct the evaluation set from the official source and evaluate their SQL predictions under the documented protocol. We will update the Reproducibility Statement with the access link, release scope, and reconstruction instructions. We also recognize the original authors' concern about training-data contamination. Developing a suitable long-term distribution approach that supports reproducibility while reducing the risk of unintended training use remains part of our follow-up work.

About the writing, thank you for the suggestion. We streamline Section 3.2 around the core findings, retaining only one or two anchor numbers and leaving the detailed values to Figure 3 and Table 6. We also update Section 3.3 to illustrate how clarification can elicit requirements left unstated in the full-information annotations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General Response

We thank the Area Chair and reviewers for their careful evaluation and constructive feedback. Our paper studies how SQL contrast helps agents identify ambiguities and use additional interaction effectively in text-to-SQL. Building on this mechanism, DuetSQL organizes clarification to improve task success while managing user effort and model computation. We compare methods under matched interaction allowances and examine their computational costs, aiming to identify effective and efficient clarification strategies suitable for practical use.

Our experiments adopt the user-simulation protocol established by BIRD-Interact (ICLR 2026). Our focus is on how clarification strategies affect interaction-budget scaling, with the solver configuration controlled in the main comparisons.

We clarify two points central to interpreting our results. First, the former “full-information” reference supplies BIRD-Interact’s complete static annotations, not an exhaustive specification of every intended business rule. Clarification permitted by the benchmark protocol can therefore elicit requirements left unstated in these annotations. We now use the term static-annotation reference and explain this distinction through its definition and a concrete case. Second, the condition described as the “bare loop” already implements SQL-contrast-guided clarification, a key technical mechanism of our approach: it uses differences among candidate SQL interpretations to identify concrete questions about user intent. Studying why this mechanism makes additional interaction useful, and how candidate-pool width changes the returns from interaction, is a central contribution of the paper. Our analysis examines when wider pools bring earlier gains and how their observed advantage narrows as more interaction becomes available. The additional stages build on this core and serve distinct accuracy and efficiency objectives.

We have made the following changes and conducted supplementary experiments:

  • Simulator checks and evaluation with another simulator family. We manually reviewed simulated dialogues from ten tasks across the compared methods. In this sample, 122/129 turns (94.6%) showed no observed leakage, and 72.9% of questions contained at least two explicit alternatives, illustrating how agents identify competing interpretations before asking users to confirm their intent. We also compared DuetSQL, ClarifyGPT, and BIRD-Interact Conv with a Qwen user simulator, holding the tasks, solver, and interaction allowance fixed. DuetSQL remained ahead of both baselines in the tested setting. The simulator also passed all UserSim-Guard thresholds under the documented evaluator configuration. The audit summary and supplementary evaluation are included in Appendix B.

  • Paired statistical analysis and independent repeated runs. We added paired analyses of key method comparisons and component ablations to assess task-success differences and resource savings. Separately, we independently reran the complete DuetSQL pipeline in a supplementary solver setting. Across these executions, DuetSQL remained ahead of the existing ClarifyGPT run in the tested setting.

  • Practical costs and response time. We supplemented token counts with solver-call counts and reference API costs to examine success–cost trade-offs across interaction budgets. The appendix also reports model-processing response-time estimates based on measured call durations, preserving DuetSQL’s evaluated workflow. We document concurrency assumptions and excluded overhead, including user-simulator generation.

  • Qualitative analysis of failures at high interaction budgets. We reviewed clarification dialogues and generated SQL for sampled failures to trace how user requirements were elicited and implemented. The review documents cases of unresolved requirements, clarified requirements not reflected in SQL, and incorrect SQL despite candidate agreement. We added a summary of this analysis to the appendix to explain why a larger interaction allowance does not necessarily resolve these failures.

  • Open research resources. We have made AMBROSIA-Interact evaluation resources available through an anonymous release to support reproducibility and further research on interactive text-to-SQL. The release includes the evaluation subset used in the paper, its databases, preprocessing and evaluation code, the interaction protocol, and the user-simulator prompt, together with instructions for reconstructing the subset from the original data. The release link and scope are documented in the Reproducibility Statement.

The manuscript now includes the reference clarification, simulator checks, response-time and qualitative failure analyses, and release information, strengthening its clarity, empirical support, and reproducibility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.