acceptodds
Under review as a conference paper at ICLR 2027

Beyond Correctness: Resolving Underspecification in Agentic Text-to-SQL

Abstract

Agentic Text-to-SQL systems can interact with users to clarify underspecified queries before generating SQL. However, a correct execution result does not necessarily imply that the agent has adequately resolved the underlying underspecification: the agent may silently make unverified assumptions that happen to match the intended answer. We show that this behavior is driven in part by premature clarification termination. Although forcing an agent to ask more questions improves execution accuracy, ambiguities are concentrated in earlier interactions, making brute-force questioning inefficient. More importantly, even when explicitly prompted to plan its clarification process, the agent frequently abandons questions that it has already identified as relevant. To address this failure mode, we introduce PlanPool, which externalizes the clarification plan as a mutable question pool. Every planned question must be explicitly asked or dropped before submission, while newly discovered ambiguities can be added during interaction. Across three benchmarks derived from BIRD-Interact and Spider, PlanPool consistently improves ambiguity coverage and reduces silent failures over unconstrained and prompt-based alternatives, while maintaining competitive execution accuracy. Our results highlight an important distinction in agentic reasoning: identifying missing information is not sufficient, and the agent must also reliably maintain and resolve it before committing to an answer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Text-to-SQL, LLM agents, underspecification, interactive

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yN7ROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies premature clarification termination in agentic Text-to-SQL: an agent may identify relevant ambiguities yet submit SQL without resolving them, sometimes obtaining the correct execution result through unverified assumptions. It proposes PlanPool, a persistent, dynamically editable clarification pool whose pending questions must be asked or explicitly dropped before submission, and evaluates it on three benchmarks derived from BIRD-Interact and Spider, reporting improved ambiguity coverage and grounded success with competitive execution accuracy.

Critical strengths

PlanPool requires no additional model training and integrates external state management with an explicit submission constraint. Within the shared evaluation harness, it improves ambiguity recall and grounded success over Plan-Prompt on all three datasets.

Critical weaknesses
  1. The groundedness metrics measure question coverage rather than resolution.

An ambiguity is counted as addressed when the simulator’s encoder maps a clarification question to its annotation (Section 2, Appendix B); whether the answer was sufficient, or was reflected in the final SQL, is not checked. The paper’s own numbers suggest that this gap is substantial. Since the four outcome categories partition the instances, Table 2 implies the unreported fourth category: "all annotated ambiguities addressed but incorrect SQL"(100 - EA - SF).

On BIRD-Full, this is 24.5% (100 - 67.1 - 8.4) for PlanPool versus 19.1%(100 - 69.8 -11.1) for Plan-Prompt: roughly one third of PlanPool’s fully covered trajectories fail, even though instances were filtered to be solvable once all annotated clarifications are provided.

GS and US are not independent: since EA = GS + US, higher GS automatically means lower US when EA stays the same. However, a correct answer without a matched clarification question is not necessarily a lucky guess. The agent may have obtained the needed information from the datasource, or the encoder may have missed a valid clarification. Interpreting US as ungrounded guessing therefore requires checking both whether user clarification was necessary and whether the question-to-annotation mapping was accurate.

  1. The experiments do not isolate the sources of improvement.

Table 3 reports unmodified pools in 65.5% of BIRD-Lite and 61.8% of BIRD-Full instances, but does not quantify the benefit of adding or dropping questions. Table 2 does not isolate the effects of explicit external state, requiring an empty pool before submission, and pool modifications. Although Section 3.1 studies fixed question budgets, PLANPOOL is not compared with baselines at matched budgets.

On BIRD-Full, PlanPool has lower observed EA than Plan-Prompt (67.1% versus 69.8%) despite asking more questions. Although Section 5.2 acknowledges that better coverage need not improve EA, a failure analysis and uncertainty estimates are needed to assess this trade-off.

Questions for authors
  1. Grounding evaluation: Table 2 implies that approximately 24.5% of all BIRD-Full instances under PlanPool have full annotation coverage but incorrect SQL. Can the authors break down these failures into incomplete clarification? Please address this question for all three datasets.

  2. Clarification necessity: Can the authors further analyze which questions require clarification to produce correct SQL and which do not affect the answer? Further discussion is possible regarding case studies and quantitative analysis.

  3. Source of performance gains: How much of PLANPOOL’s improvement comes from its pool mechanism rather than asking more questions? Can the authors compare it with Plan-Prompt and fixed-budget baselines using the same question budget?

Reviewer_gJAkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates underspecification in interactive Text-to-SQL: user requests may admit multiple reasonable interpretations, and a SQL query that produces the correct result does not necessarily indicate that the system has actually resolved these ambiguities.

The authors propose PlanPool, which maintains pending clarification questions in an external, modifiable question pool. It supports adding, removing, and asking questions, and permits submission of the final SQL query only when no pending questions remain in the pool. The paper also uses execution accuracy (EA), ambiguity coverage (Recall), Grounded Success (GS), and Silent Failure (SF) to distinguish between result correctness and whether the interaction process covers the annotated ambiguities.

Experiments are conducted on BIRD-Lite, BIRD-Full, and interactive tasks constructed from Spider. Compared with Plan-Prompt, PlanPool improves Recall and GS on all three evaluation sets, but its effect on EA is mixed: on BIRD-Full, EA decreases from 69.8% to 67.1% (p. 7, Table 2).

Critical strengths
  1. The problem is important, and the evaluation perspective is valuable.

    Section 2 (p. 3) explicitly distinguishes execution success from ambiguity coverage, with GS requiring both. This helps reveal cases where “the answer happens to be correct, but the interaction has not elicited the necessary information,” adding a useful diagnostic dimension to agentic Text-to-SQL. The value of this contribution does not depend on the authors claiming to be the first to introduce interactive clarification.

  2. The mechanism is clear and can accommodate changes to the plan during interaction.

    The method description (p. 5) and the prompts (pp. 21–22) allow readers to inspect the operations on the question pool. The appendix example shows how, after the interpretation of CPI is established, the now-irrelevant ERF question is removed and a question about the CPI formula is added (pp. 16–18, especially Table 6 on p. 17). This supports the rationale for updating clarification plans in response to answers, showing that the mechanism goes beyond simply repackaging a fixed checklist.

  3. Improvements in coverage metrics are consistent across datasets, and the authors also report costs and negative results.

    Table 2 reports the following changes from Plan-Prompt to PlanPool:

    DatasetEARecallGSAverage questions asked
    BIRD-Lite79.5 → 81.380.8 → 87.542.1 → 49.13.3 → 4.2
    BIRD-Full69.8 → 67.178.8 → 84.645.8 → 48.02.9 → 3.6
    Spider75.4 → 76.193.2 → 94.369.0 → 70.72.2 → 2.5

    The first three metric columns—EA, Recall, and GS—are expressed as percentages. Table 4 (p. 9) also reports token usage and monetary costs, allowing readers to see that the improvements in coverage come with additional interaction and computational overhead.

Critical weaknesses

1. The current evidence does not adequately separate the effects of the dynamic question pool from those of asking more questions and enforcing plan completion. — Insufficient evidence for attributing gains to the core mechanism. Table 2 shows that PlanPool increases the average number of questions asked on all three evaluation sets. Figure 2 (p. 4) already shows that increasing the target number of questions can itself substantially improve EA. At the same time, PlanPool changes the external state representation, completion constraints, and add/drop capabilities.

2. GS is a useful operational metric, but a validation gap remains between “asking about an ambiguity” and “resolving it.” — Metric validity concern. Page 3 operationalizes addressing an ambiguity as asking the corresponding clarification question, and GS combines this criterion with EA. There is no mathematical error in the definition itself, but the definition alone does not establish that the answer is sufficiently clear, that the final SQL correctly incorporates it, or that a deleted question is indeed no longer necessary.

3. Uncertainty around the smaller gains and the scope of the conclusions need to be clearer. — Strength of evidence and limits to generalization. The sample sizes in Table 1 (p. 6) are 171, 225, and 297, respectively; the GS gains in Table 2 are 7.0, 2.2, and 1.7 percentage points. Based on the rounded rates, these correspond to approximate net differences of 12, 5, and 5 successful cases, respectively. These are not counts of errors corrected in paired comparisons, nor do they constitute a significance test. The main results table does not present uncertainty estimates based on paired comparisons. Section 5.1 (p. 6) and Appendix A.1 (p. 13) also indicate that the BIRD evaluation tasks were filtered for solvability with oracle clarification.

Questions for authors

With comparable actual numbers of questions asked, how much of an advantage in GS and Recall does PlanPool retain over a static checklist that must be completed? If these advantages persist, this would substantially strengthen the methodological contribution.

Reviewer_yKxiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies premature clarification termination in agentic Text-to-SQL: an agent may identify ambiguities but submit SQL before resolving them. PlanPool stores clarification questions in an external pool. The agent can ask, add, or drop questions, and submit SQL only when the pool is empty. Results on three benchmarks show better ambiguity coverage and grounded success, but execution accuracy does not consistently improve.

Critical strengths

S1: The paper makes a useful distinction between execution correctness and resolving user intent. A correct result can still rely on unverified assumptions.

S2: The question-budget study shows that asking more questions improves accuracy, but later questions often have less value. This gives useful evidence about clarification behavior.

S3: PlanPool is simple and easy to understand. The evaluation reports both groundedness and interaction cost, giving a fuller picture than execution accuracy alone.

Critical weaknesses

W1: The method is mainly a question checklist managed by the LLM. SQL submission is allowed only when the pool is empty. The design is simple, and ablations are needed to show the value of dynamic updates and the submission constraint.

W2: PlanPool uses more input tokens and asks more questions than Plan-Prompt, but execution accuracy does not consistently improve. Equal-budget comparisons and an evaluation of user effort are missing.

W3: The paper lacks comparisons with directly related methods such as AmbiSQL (Ding et al., 2025). It should also include baselines with the same question budget. A practical agent such as Codex in Plan mode would help test its real-world value.

W4: AmbiSQL (Ding et al., 2025) already uses iterative clarification before SQL generation, and FlowAgent (Shi et al., 2025) uses external controllers to enforce workflow rules. The paper does not clearly explain what the question pool adds beyond these existing designs.

Questions for authors

Q1: Can you ablate dynamic pool updates and the submission constraint separately? How does PlanPool compare with a static checklist that must be completed before SQL submission?

Q2: Can you compare with AmbiSQL (Ding et al., 2025) under the same interaction setting? A practical agent such as Codex in Plan mode would also help assess its real-world value.

Q3: What is the main technical difference from AmbiSQL’s iterative clarification and FlowAgent’s external workflow control? Beyond applying these ideas to a question pool, what new mechanism does PlanPool introduce?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.